Sunday, April 23, 2017

OSI model layered approach to CRTC policy required to get quality of service

Re: Telecom Notice of Consultation CRTC 2017-49: Review of the competitor quality of service regime

While the consultation is in the form of a series of questions, my intervention requires context.

While necessary at this time, I consider the "competitor quality of service regime" to be a transition policy to deal with problems caused by the way we managed convergence in Canada.  My recent submission to the Canadian content in a digital world consultations was focused on convergence policy.

If we are to solve the underlying problems on a more permanent basis we need to adopt policy that mirrors the OSI model layered approach of the underlying digital communications technology. The digital transition involved moving from purpose-built networks to a layered approach, and our regulations should have adapted from regulating as if the underlying technology were still purpose-built.

In short, this would mean:
  • Municipal layers 1 and 2 (or rather, what is now often called layer 2.5) should be treated as a utility and be owned and managed by different levels of government as is the case for other utilities.  While I will use the term "last mile" as a short-form, this is a municipal network that shouldn't require private sector entities to make multiple connections to the same municipality.
  • All services built "over the top" on layers 3 and above should exist in a competitive private sector marketplace.
The problem with trying to discuss a "competitor quality of service regime" is that incumbent last mile providers are providing layers 1-2.5  and claiming their own "over the top" (layer 3+) services should be regulated differently than other companies "over the top" (layer 3+) services.   They have re-defined the concept of "over the top" in an anti-competitive manner to mean services of competitors, rather than adopting neutral technology-driven language.

I made an intervention in front of the CRTC on 2009-12-09 to make these points.  I live and work in Ottawa, and can be available to discuss this with commissioners and staff at the CRTC in a formal or informal setting.

Q1. How has the current Q of S regime performed in fostering competition?

While the current regime might have disallowed private-sector last-mile providers from completely wiping out businesses built on layers 3+, I do not believe it has fostered competition.

Q2. Are market forces sufficient to ensure a high level of service or are Q of S regulatory measures required?

Market forces cannot function unless the market is correctly defined.

When a new retailer opens it does not have to get "permission" from a competing retailer for its customers to use the roads required to get to the retailer.  If Walmart claimed ownership of all the roads in a specific town we would never suggest that "market forces" were sufficient for any public policy goal.

Q3. If a competitor Q of S regime is required, what should its objectives be?

We should be trying to migrate from the legacy analog-era regulatory regimes to a digital layered approach.  The objectives of the Q of S regime should be to seek to achieve in the current vertically integrated marketplace conditions as close to those that would be possible if we had structural separation.

Customers of different layer 3+ services should not be getting different treatment from layer 1-2.5 providers.    As long as last mile providers are allowed to also provide layer 3+ services, strongly enforced regulations will be required.

Q4. If other regulatory measures are required, what should they include?

The problems I am identifying are not specific to the Q of S regime, but all areas of CRTC policy (Broadcast and Telecom Acts) where the lack of structural separation has an ongoing impact.

My intervention in front of the CRTC in 2009 was in the context of local television, as I see what is traditionally called "Cable Television" in the analog era to be yet another layer 3+ service in the digital era.  While many more people have "cut the chord" (transitioning from analog-era video services) than 2009, my intervention still applies to the current situation.


These questions relate to the rate rebate plan (RRP). As long as last mile providers are allowed to also sell layer 3+ services, financial as well as other incentives to provide good services to all layer 3+ companies is required.  This will require both carrot and stick, as vertically integrated providers will always want to privileged their own branded services.

Q9. What criteria should be used to select the types of services, if any, that should be included in the competitor Q of S regime?

All layer 3+ services should be included in the Q of S regime.

We need to transition from thinking of IPv4 and IPv6 with publicly routed addresses as being "special" compared to other protocols or private network addressing.

There is no legitimate reason why only Bell should be allowed to provide a "Fibe TV" style service. Other BDUs should be able to form to provide those services and directly compete with the services of any existing layer 1-2.5 provider.  We need to break the current oligopoly which, among other things, includes inappropriate policy in Bill C-11 from 2002 which defined  “new media retransmitter” in such a way as to disallow layer 3+ companies which weren't also last mile layer 1-2.5 providers.

Q10. What specific services should be subject to the regime?

What is called “Wholesale Network Access” should be the bare minimum immediately, but this should quickly expand to all layer3+ services.

While not explicitly part of this consultation, I also support CNOC's proposal to apply the Wholesale Network Access regime to fiber to the premises.  We have experienced this problem within our family where my parents-in-law moved into a condo where the condo corporation had already made a deal with a specific vertically integrated provider.  My father-in-law was unable to move his existing Internet service (with Teksavvy) to his new home, and there is no competition within that building.

Q11. Should the competitor Q of S regime be expanded to include cable carriers, wireless carriers, small ILECs, and Northwestel? If so, should the competitor Q of S regime apply to all such carriers or should there be a threshold based on the size of the company and/or other factors?

All private sector last mile wired or wireless layer 1-2.5 providers should be under Q of S.

Exceptions can be negotiated for specific public sector entities, such as very small municipalities who might not have the resources to provide quality of service for other than a bare minimum of communications services.

Northwestel is owned by Bell Canada, and need not be listed separately from other private sector entities.  Regional governments should have an incentive to own their own networks, and private sector providers being under more rigorous regulation should be one of those incentives.

Q12. Does the complaint-based system continue to be the best means of monitoring Q of S for small ILECs?

Most customers of communications services are not aware of who they should complain to.  Then a last-mile provider provides poor service it is the layer 3+ provider that often gets blamed.  The onus should not be on communications customers to understand these different layers, but for the regulator to work closely with layer 3+ providers to adequately regulate layer 1-2.5 providers.


Further questions are at a level of detail that goes beyond the focus of this intervention, so I will leave unanswered.

Wednesday, April 5, 2017

Next steps for the platform: IIIF, JHOVE

This is a new fiscal year.  I'm excited to discuss some of our current plans, and hopefully get some feedback and help from the community.

Some related Internal Projects for our 3 person DevOps team include adoption of IIIF APIs, JHOVE, and updates to associated METS application profile and schemas.

A quick note about when?

The Phoenix Project describes how IT Operations only do 4 things. This work falls under Internal Projects, and will be bumped by Business Projects as well as Unplanned Work. While this work may be exciting to the DevOps team, other work will take precedence.  This means that while I can say we are working on it, we can't offer any useful idea of when there will be something other people can use.

International Image Interoperability Framework

The name gives an impression much less than what this community is doing.  It isn't only a common way to access images, but a community that is working on common APIs and documentation for institutions like ours which offer access to images.  When I first glanced I thought this was a possible replacement for part of our Content Server (See: The Canadiana preservation network and access platform), but it turns out to be a community defining interoperability between most aspects of our access platform.

Adoption will come in stages, with the replacement of our Content Server being the first step.  The content server can be thought of as having 3 parts:
  • Authentication
  • Authenticated direct access to files
  • Authenticated access to derivatives of images


We have decided to adopt Cantaloupe, which provides a simple configuration method to handle our authentication and access to images in our TDR (Trustworthy Digital Repository).  We could have enhanced our own image server to handle IIIF, but we felt it better if we adopted and participated in an existing community.  We will be retrofitting our existing access services to use the IIIF image server and decommission our existing server.

Setting up a test Cantaloupe server with access to TDR files involved writing a small self.get_pathname(identifier)  Ruby function to return the correct filesystem pathname based on an identifier.  What will take more work is the authorization function to denying access to files we aren't able to offer access to.

IIIF Authentication API

The authentication model that IIIF uses is different than what we have been using, so we'll need to adjust.  Our existing authorization token was presumed to be unique to accessing content within an AIP (See: OAIS) within our TDR, and a new token would be requested if the client needed access to a new AIP.  We will be adopting a different token, and require a database lookup to confirm the AIP being accessed is part of a collection the patron is authorized to access.  While most AIPs are sponsored, not all are and thus not every patron will have access to every AIP.

Direct access to files

Once we have the first two in place, all we need to do for direct access is verify credentials, find the base path for the AIP in the disk pools, and allow the HTTP server to send the file to the client.

IIIF Presentation API

Reading the API documentation reminded us of many conversations at our whiteboard. Our current collection management interface is simple: a collection is a tag that is attached to an AIP ID within a non-TDR database.

We planned to move to collections being a list which can contain other collections or AIPs, which is the model IIIF uses.  Specific collections would be able to be purchased, and for these we would compile that to tags to allow for a quick reverse-lookup for authentication.

The same is true of other aspects of the Presentation API that map to expansion ideas we have had for a few years.

Our current (2012 edition) AIP and SIP documents describe a layout and a METS profile that was designed for the ECO project.

  • Set of scanned images (JPEG, JPEG2000, TIFF)
  • Single down-loadable PDF, usually derived from OCR of all the images
  • Single "physical" order for the images  (METS structMap, IIIF Sequence)
There are a number of enhancements to our AIP structure we have been discussing, and each fit well within the IIIF presentation API

  • Allowing multiple structMaps to define multiple orders, such as when image corrections are made (pages removed, or added), or the order wasn't correct. One structMap can describe all the images within the AIP, but other named structMaps(sequences) can describe a separate sequence for display.
  • Storing the "master" as well as other derived formats (other image formats, ALTO XML OCR data, OCR derived PDF) associated with that master, making the relationship between the master image and the derivatives clear.

We had a prototype a summer student created for us of a tool which was intended to edit those structMaps.  Now that we are moving towards IIIF we are most likely to adopt using  an IIIF Manifest Editor instead, providing mechanisms for IIIF Manifests to be used by our AIP manipulation software to generate METS structMaps.

Adoption of IIIF will impact most aspects of our platform, making full integration into many incremental projects.


It all started with IIIF Manifests needing the height and width dimensions for canvas and image resources.  This is metadata our lead software developer has wanted the front end interface to have access to for some time, and our metadata architect had planned to look into MIX (NISO Metadata for Images in XML) as a way to store this information in our METS records.  The MIX documentation references JHOVE, and JHOVE has been mentioned to us quite often over the years.  It was in our list of tools to investigate adopting.

In our TDR we have tens of millions of files, with some of the earliest scanned and OCR'd in the late 1990's.  As part of the CRL TDR certification we added checks for new files being added to our TDR (ImageMagick's identify for our JPEG, JPEG2000, TIFF and PDF files).  Prior to that our guarantee was based on our use of Bagit for AIPs and revisions, which was that if any change happened to the file in the TDR (bit rot, etc) that we would detect it with our constant re-validation of the MD5's of all files in all AIPs.  We always keep copies of files in multiple physical server rooms across the country.  During content ingest we would also confirm our METS record referenced all the files being submitted.   While we wanted to adopt something more robust such as JHOVE for identification and validation of the format of files, we weren't able to allocate the resources at the time to implement.

On March 24 I downloaded the latest JHOVE (XML files generated indicate release="1.16.5") and set it to generate an XML file for all of our files.  This process is ongoing, with 60 million files categorized and counting.

I expected to find problems with our earliest files, but was surprised to find issues reported from our most recent additions.  We would like to have JHOVE file identification/validation as part of our ingest process, only adding new files which are recognized.  Before we can do this we need to work out compatibility issues.

If you have adopted JHOVE, and manipulate PDF files as part of your processing, I am curious if you have seen the problem we found when joining single-page PDFs into multi-page PDFs using Poppler (via Ubuntu) or PDFTK (via Ubuntu). It is surprising that JHOVE is finding errors for multi-page PDF files generated with these common utilities.