Sunday, April 23, 2017

OSI model layered approach to CRTC policy required to get quality of service

Re: Telecom Notice of Consultation CRTC 2017-49: Review of the competitor quality of service regime

While the consultation is in the form of a series of questions, my intervention requires context.

While necessary at this time, I consider the "competitor quality of service regime" to be a transition policy to deal with problems caused by the way we managed convergence in Canada.  My recent submission to the Canadian content in a digital world consultations was focused on convergence policy.

If we are to solve the underlying problems on a more permanent basis we need to adopt policy that mirrors the OSI model layered approach of the underlying digital communications technology. The digital transition involved moving from purpose-built networks to a layered approach, and our regulations should have adapted from regulating as if the underlying technology were still purpose-built.

In short, this would mean:
  • Municipal layers 1 and 2 (or rather, what is now often called layer 2.5) should be treated as a utility and be owned and managed by different levels of government as is the case for other utilities.  While I will use the term "last mile" as a short-form, this is a municipal network that shouldn't require private sector entities to make multiple connections to the same municipality.
  • All services built "over the top" on layers 3 and above should exist in a competitive private sector marketplace.
The problem with trying to discuss a "competitor quality of service regime" is that incumbent last mile providers are providing layers 1-2.5  and claiming their own "over the top" (layer 3+) services should be regulated differently than other companies "over the top" (layer 3+) services.   They have re-defined the concept of "over the top" in an anti-competitive manner to mean services of competitors, rather than adopting neutral technology-driven language.

I made an intervention in front of the CRTC on 2009-12-09 to make these points.  I live and work in Ottawa, and can be available to discuss this with commissioners and staff at the CRTC in a formal or informal setting.

Q1. How has the current Q of S regime performed in fostering competition?

While the current regime might have disallowed private-sector last-mile providers from completely wiping out businesses built on layers 3+, I do not believe it has fostered competition.

Q2. Are market forces sufficient to ensure a high level of service or are Q of S regulatory measures required?

Market forces cannot function unless the market is correctly defined.

When a new retailer opens it does not have to get "permission" from a competing retailer for its customers to use the roads required to get to the retailer.  If Walmart claimed ownership of all the roads in a specific town we would never suggest that "market forces" were sufficient for any public policy goal.

Q3. If a competitor Q of S regime is required, what should its objectives be?

We should be trying to migrate from the legacy analog-era regulatory regimes to a digital layered approach.  The objectives of the Q of S regime should be to seek to achieve in the current vertically integrated marketplace conditions as close to those that would be possible if we had structural separation.

Customers of different layer 3+ services should not be getting different treatment from layer 1-2.5 providers.    As long as last mile providers are allowed to also provide layer 3+ services, strongly enforced regulations will be required.

Q4. If other regulatory measures are required, what should they include?

The problems I am identifying are not specific to the Q of S regime, but all areas of CRTC policy (Broadcast and Telecom Acts) where the lack of structural separation has an ongoing impact.

My intervention in front of the CRTC in 2009 was in the context of local television, as I see what is traditionally called "Cable Television" in the analog era to be yet another layer 3+ service in the digital era.  While many more people have "cut the chord" (transitioning from analog-era video services) than 2009, my intervention still applies to the current situation.


These questions relate to the rate rebate plan (RRP). As long as last mile providers are allowed to also sell layer 3+ services, financial as well as other incentives to provide good services to all layer 3+ companies is required.  This will require both carrot and stick, as vertically integrated providers will always want to privileged their own branded services.

Q9. What criteria should be used to select the types of services, if any, that should be included in the competitor Q of S regime?

All layer 3+ services should be included in the Q of S regime.

We need to transition from thinking of IPv4 and IPv6 with publicly routed addresses as being "special" compared to other protocols or private network addressing.

There is no legitimate reason why only Bell should be allowed to provide a "Fibe TV" style service. Other BDUs should be able to form to provide those services and directly compete with the services of any existing layer 1-2.5 provider.  We need to break the current oligopoly which, among other things, includes inappropriate policy in Bill C-11 from 2002 which defined  “new media retransmitter” in such a way as to disallow layer 3+ companies which weren't also last mile layer 1-2.5 providers.

Q10. What specific services should be subject to the regime?

What is called “Wholesale Network Access” should be the bare minimum immediately, but this should quickly expand to all layer3+ services.

While not explicitly part of this consultation, I also support CNOC's proposal to apply the Wholesale Network Access regime to fiber to the premises.  We have experienced this problem within our family where my parents-in-law moved into a condo where the condo corporation had already made a deal with a specific vertically integrated provider.  My father-in-law was unable to move his existing Internet service (with Teksavvy) to his new home, and there is no competition within that building.

Q11. Should the competitor Q of S regime be expanded to include cable carriers, wireless carriers, small ILECs, and Northwestel? If so, should the competitor Q of S regime apply to all such carriers or should there be a threshold based on the size of the company and/or other factors?

All private sector last mile wired or wireless layer 1-2.5 providers should be under Q of S.

Exceptions can be negotiated for specific public sector entities, such as very small municipalities who might not have the resources to provide quality of service for other than a bare minimum of communications services.

Northwestel is owned by Bell Canada, and need not be listed separately from other private sector entities.  Regional governments should have an incentive to own their own networks, and private sector providers being under more rigorous regulation should be one of those incentives.

Q12. Does the complaint-based system continue to be the best means of monitoring Q of S for small ILECs?

Most customers of communications services are not aware of who they should complain to.  Then a last-mile provider provides poor service it is the layer 3+ provider that often gets blamed.  The onus should not be on communications customers to understand these different layers, but for the regulator to work closely with layer 3+ providers to adequately regulate layer 1-2.5 providers.


Further questions are at a level of detail that goes beyond the focus of this intervention, so I will leave unanswered.

Wednesday, April 5, 2017

Next steps for the platform: IIIF, JHOVE

This is a new fiscal year.  I'm excited to discuss some of our current plans, and hopefully get some feedback and help from the community.

Some related Internal Projects for our 3 person DevOps team include adoption of IIIF APIs, JHOVE, and updates to associated METS application profile and schemas.

A quick note about when?

The Phoenix Project describes how IT Operations only do 4 things. This work falls under Internal Projects, and will be bumped by Business Projects as well as Unplanned Work. While this work may be exciting to the DevOps team, other work will take precedence.  This means that while I can say we are working on it, we can't offer any useful idea of when there will be something other people can use.

International Image Interoperability Framework

The name gives an impression much less than what this community is doing.  It isn't only a common way to access images, but a community that is working on common APIs and documentation for institutions like ours which offer access to images.  When I first glanced I thought this was a possible replacement for part of our Content Server (See: The Canadiana preservation network and access platform), but it turns out to be a community defining interoperability between most aspects of our access platform.

Adoption will come in stages, with the replacement of our Content Server being the first step.  The content server can be thought of as having 3 parts:
  • Authentication
  • Authenticated direct access to files
  • Authenticated access to derivatives of images


We have decided to adopt Cantaloupe, which provides a simple configuration method to handle our authentication and access to images in our TDR (Trustworthy Digital Repository).  We could have enhanced our own image server to handle IIIF, but we felt it better if we adopted and participated in an existing community.  We will be retrofitting our existing access services to use the IIIF image server and decommission our existing server.

Setting up a test Cantaloupe server with access to TDR files involved writing a small self.get_pathname(identifier)  Ruby function to return the correct filesystem pathname based on an identifier.  What will take more work is the authorization function to denying access to files we aren't able to offer access to.

IIIF Authentication API

The authentication model that IIIF uses is different than what we have been using, so we'll need to adjust.  Our existing authorization token was presumed to be unique to accessing content within an AIP (See: OAIS) within our TDR, and a new token would be requested if the client needed access to a new AIP.  We will be adopting a different token, and require a database lookup to confirm the AIP being accessed is part of a collection the patron is authorized to access.  While most AIPs are sponsored, not all are and thus not every patron will have access to every AIP.

Direct access to files

Once we have the first two in place, all we need to do for direct access is verify credentials, find the base path for the AIP in the disk pools, and allow the HTTP server to send the file to the client.

IIIF Presentation API

Reading the API documentation reminded us of many conversations at our whiteboard. Our current collection management interface is simple: a collection is a tag that is attached to an AIP ID within a non-TDR database.

We planned to move to collections being a list which can contain other collections or AIPs, which is the model IIIF uses.  Specific collections would be able to be purchased, and for these we would compile that to tags to allow for a quick reverse-lookup for authentication.

The same is true of other aspects of the Presentation API that map to expansion ideas we have had for a few years.

Our current (2012 edition) AIP and SIP documents describe a layout and a METS profile that was designed for the ECO project.

  • Set of scanned images (JPEG, JPEG2000, TIFF)
  • Single down-loadable PDF, usually derived from OCR of all the images
  • Single "physical" order for the images  (METS structMap, IIIF Sequence)
There are a number of enhancements to our AIP structure we have been discussing, and each fit well within the IIIF presentation API

  • Allowing multiple structMaps to define multiple orders, such as when image corrections are made (pages removed, or added), or the order wasn't correct. One structMap can describe all the images within the AIP, but other named structMaps(sequences) can describe a separate sequence for display.
  • Storing the "master" as well as other derived formats (other image formats, ALTO XML OCR data, OCR derived PDF) associated with that master, making the relationship between the master image and the derivatives clear.

We had a prototype a summer student created for us of a tool which was intended to edit those structMaps.  Now that we are moving towards IIIF we are most likely to adopt using  an IIIF Manifest Editor instead, providing mechanisms for IIIF Manifests to be used by our AIP manipulation software to generate METS structMaps.

Adoption of IIIF will impact most aspects of our platform, making full integration into many incremental projects.


It all started with IIIF Manifests needing the height and width dimensions for canvas and image resources.  This is metadata our lead software developer has wanted the front end interface to have access to for some time, and our metadata architect had planned to look into MIX (NISO Metadata for Images in XML) as a way to store this information in our METS records.  The MIX documentation references JHOVE, and JHOVE has been mentioned to us quite often over the years.  It was in our list of tools to investigate adopting.

In our TDR we have tens of millions of files, with some of the earliest scanned and OCR'd in the late 1990's.  As part of the CRL TDR certification we added checks for new files being added to our TDR (ImageMagick's identify for our JPEG, JPEG2000, TIFF and PDF files).  Prior to that our guarantee was based on our use of Bagit for AIPs and revisions, which was that if any change happened to the file in the TDR (bit rot, etc) that we would detect it with our constant re-validation of the MD5's of all files in all AIPs.  We always keep copies of files in multiple physical server rooms across the country.  During content ingest we would also confirm our METS record referenced all the files being submitted.   While we wanted to adopt something more robust such as JHOVE for identification and validation of the format of files, we weren't able to allocate the resources at the time to implement.

On March 24 I downloaded the latest JHOVE (XML files generated indicate release="1.16.5") and set it to generate an XML file for all of our files.  This process is ongoing, with 60 million files categorized and counting.

I expected to find problems with our earliest files, but was surprised to find issues reported from our most recent additions.  We would like to have JHOVE file identification/validation as part of our ingest process, only adding new files which are recognized.  Before we can do this we need to work out compatibility issues.

If you have adopted JHOVE, and manipulate PDF files as part of your processing, I am curious if you have seen the problem we found when joining single-page PDFs into multi-page PDFs using Poppler (via Ubuntu) or PDFTK (via Ubuntu). It is surprising that JHOVE is finding errors for multi-page PDF files generated with these common utilities.

Friday, February 24, 2017

Why I joined the CPC to vote for Michael Chong

I have to admit that when I first heard that Michael Chong was running for the leadership of the Conservative Party of Canada (CPC) I thought: Oh, cute - a progressive conservative is running -- they'll squash him like a bug.

I had an opportunity to hear Mr. Chong in person on Wednesday February 22 at an event hosted at the offices of OpenConcept (a business owned by long-time friend Michael Gifford).  There was about 30 guests there, and Mr. Chong's meet-and-greeting very quickly turned into a Q&A session.

Most of what I heard I really liked.  Enough to decide to become a member of the party in order to vote in the leadership election in May (March 28 is the final date to become a member if you want to have your say).  I also made a $350 campaign contribution to Mr. Chong.

I'll highlight three areas of policy he spoke about, the first being from that bulk of ideas I agreed with, then one area I am mostly with him but with some reservations, and one area I disagreed.  Two are key elements of his platform: Economic Opportunity, Environmental Opportunity, and Democratic Opportunity.

Green Tax Policy

Mr. Chong is a GreenPAC endorsed candidate, and I was grinning from ear-to-ear when he was talking about his proposal for a revenue neutral shift from income taxes to a carbon tax (one much higher than proposed by the Liberals).  He even used the phrase "taxing things which are bad rather than things which are good", a phrase that has been used by the Green Tax Shift movement for decades.  (The shortform "Tax bads, not goods" confuses people as they think of the wrong meaning of "goods").

I have been a large supporter of the Green Tax Shift since I heard about it in the early 1990's.  Then I was a member of the Green Party, and remember one election where we were setting up a campaign office for the region (we didn't have the funding to have per-district offices).  This was a rental of a building where the previous  tenant was "H&R Block Income Tax".  I wanted us to put up a sign that said "Abolish" over the "H&R Block" part of the sign, but was opposed by some in the party which had more traditional lefty views on these issues (one of the first signs I might have been in the wrong party).

The question came up about why the carbon tax shouldn't be put into general revenue and used to increase fiscal capacity, reduce deficits/debt, etc.   His answer was that he wanted those other issues to be separate, and that this should be a revenue neutral shift so that the average taxpayer wouldn't be opposed to the tax (the average person wouldn't feel a tax increase, while those who pollute more than average would be paying more than those of us who pollute lower than average would be paying less).

The benefits of this policy are greater than he had time to discuss.  When tax-shifting you are taking something that is out of your hands and putting it into your control.  If you want to do something to reduce your taxes you can make better choices for your energy usage and other activities which are carbon emitters.  It becomes both carrot and stick, which is far more efficient than a stick-only approach that will meet with large political opposition from a wide variety of citizens.

Mr. Chong discussed how income tax is an inefficient tax, and that its introduction was only a temporary measure to help pay for Canada's involvement in World War 1.  While the last surviving war veteran from WW1 died in 2010, income tax is unfortunately still here.  To be clear, I'm not personally against taxes or government, but I think taxing income is the wrong way to go.

Electoral Reform

The question came up about where he stood on electoral reform.  He said that most of the proposals to replace our current voting system had some seats granted to political parties, and that we needed to reform political parties first before considering these types of changes.

I understand where he is coming from, but am disappointed with fellow Canadians that this is the answer.  He is correct in that nearly all the proposals that have come from the most vocal activist groups are party functioning, granting "top up" seats based on implied or explicit support for political parties.

A different direction could be taken, which is to take the party-focused features (Like optimizing for Gallagher Index style proportionality) off the table.  Ranked ballots in single and/or multi-member districts without top-up seats or above-the-line voting doesn't privilege party affiliation.  Moving from single-member districts to multi-member districts enables competition between candidates from the same party within a multi-member district, further diminishing the influence of political parties.

Unfortunately I can't fault Mr Chong for his views. If we went forward with a referendum today (maybe a ranked ballot, like was used in PEI and is being used to decide the Conservative Party leader), it is quite likely that some party-privileging system would be brought in and the excessive control over MPs by the party and party leader would only become worse.

We not only need to do as Mr. Chong suggests and reform parties, as well as both the upper and lower houses, we need to help educate fellow Canadians about the features of different voting systems and the risks associated with some of these features.

Merging government departments and agencies

This is one of those areas of policy that a few attendees had a problem with.

Mr Chong acknowledged that government shouldn't be treated like the private sector, and that the PM should not be thought of like the CEO.

He then proceeded to use private sector examples to explain why merging government departments would be a good idea.

There are many differences between the public and private sector, and this is one of those areas where the differences need to be recognized.   In a company, the company will work best when everyone is moving in the same direction.  If some branch thought the company was moving the wrong direction they could split off and form their own company that could compete or cooperate with the original company.

Governments don't work that way.  There are different departments with bureaucracies that have competing mindsets, and that is a good thing.  There is a monopoly on governance, and some branch that thinks another branch is heading the wrong direction can't split off and form their own government.  They need to hash it out within the structure of government, with there being debate between government departments that is not unlike the debate that happens between MPs in parliament.

While I believe there are efficiencies to be had in government, I believe merging dissimilar departments or even branches is harmful.  I still believe it is wrong for consumer affairs to have been added to Industry Canada (Now Innovation, Science and Economic Development) as that creates a conflict of interest within the department that would have been easier to negotiate if it were different departments (and thus different management structure trying to bring forward policy initiatives).

My political context

When I became politically active in the early 1990's I was introduced to the Green Party by someone who was familiar with the center-right German Greens. The Greens resonated more with me when the NDP was doing well (and lefties headed to the NDP), but I felt out-of-place whenever the NDP was doing poorly (and more lefties came to the Green Party).

I joined the Progressive Conservative party in 1998 because I was excited about the leadership campaign.  What David Orchard was saying resonated with me, and the party as a whole was more appealing the more the aspects of the conservative movement that didn't resonate with me moved to the Reform party.  I was called a tourist by Joe Clark during the campaign, and after that race which Joe Clark won I didn't really feel welcome.  Even though Joan Russow was leader, and her ideas didn't resonate with me at all, I returned to the Green Party as I believed at the time I had to be involved in a party to be involved in politics.

Starting in 2002 I started to meet more sitting MPs, and the more MPs I met the less partisan I became.  There were days when I would walk out of the office of an NDP MP and into the office of a Conservative MP, and it did not seem politically jarring at all.

I currently don't consider myself partisan, and find that there are candidates and MPs from every party who I share many ideas with, and well as MPs from each party which I don't share many ideas with.   I have found that political affiliation is a poor indicator of an MPs views on the issues that concern me the most.

While I still believe I'm politically center-right, I don't think I will ever feel at home exclusively with any party.  I suspect instead I'll want to become more involved with the individuals within the parties that I support.

Even from listening to Mr Chong for a very short amount of time on Wednesday evening I believe he is one of those individuals I can support -- and will be supporting as he runs for leadership of one of the two parties whose leader is most likely to become the next Prime Minister.

Whether or not Mr Chong wins, a good sized showing will demonstrate that there is support for the type of conservatism that he represents!  Whoever becomes the next CPC leader will need to listen more closely to these views.

Saturday, February 11, 2017

More than 25 years of Free Software

One of the most critical discoveries that determined the form that my business and advocacy work would later take was the discovery of the Free Software Foundation and the GNU Project.

While I do not have an exact date of when I joined the gnu.misc.discuss newsgroup, a search on Google indicates that my first posting was on Feb 11, 1992. The topic was: Public Domain re-worked and re-released under GNU Public License?. It is interesting to go back and time and note that I have learned quite a bit about software licenses and copyright law since that time. I think my English and spelling have improved as well.

(Re-post from 2003 article on Digital Copyright Canada. Most of my recent contributions have been to this blog)

Why a long-time electoral reformer will be staying clear of the "Day of Action for Electoral Reform"

A few partisan and multi-partisan groups including the NDP, Green Party (provincial and federal), and Fair Vote Canada have sent out announcements to their members about a "National Day of Action for Electoral Reform".  While I've been an active electoral reformer for many decades, I will be avoiding this event.

Many of the reasons why were articulated in an earlier article on how Electoral Modernization is hard, but worth it.   It discussed how compromise and consideration of other viewpoints was critical to move forward.

When Justin Trudeau and his election platform team stated that, "We are committed to ensuring that 2015 will be the last federal election conducted under the first-past-the-post voting system" they likely envisioned a specific campaign.  On one side would be those who wanted to keep the current First Past the Post (FPTP) system either because it benefit them or they were simply resistant to any change.   On the other side would be a broad coalition of people who saw the many flaws in the current system and had a proposal to move forward with a solution to the major flaws.

While these multi-partisans like to use the phrase "voter equality", this was not the campaign they ran. They picked one specific flaw in the current system that has meaning to partisans (if the seats in parliament are allocated to parties in proportion to alleged votes for parties), and dismissed all other concerns as being irrelevant as well as dismissing the people who had those concerns.   This splintered those who wanted change into multiple camps, and made a broad coalition for change impossible.

With a broad coalition not possible, Justin Trudeau and the Liberal Government correctly concluded there is no consensus on this issue and was forced to back away from their election promise.

In January 2013 I cancelled my Fair Vote Canada membership and donation. I felt a need to distance myself from FVC for the sake of the electoral modernization movement in Canada.

FPTP has many flaws, and many believe vote splitting among multiple candidates that voters would be happy with is the largest flaw.  The only solution to that particular problem is ranked ballots, so it should be recognized that proposals for any system that doesn't involve ranked ballots is not a proposal aimed at "voter equality" or "fair voting".

FVC's campaign against ranked ballots in municipal elections was expected to cause confusion. Most municipalities do not have political parties, so the question of whether the makeup of city council is allocated proportionally to parties based on alleged votes for parties is an irrelevant concept.  If ranked ballots were supposedly bad in municipalities, then reformers who were repeating talking points would also be claiming that ranked ballots would be inappropriate in all elections.  The distinction between single-member and multi-member districts would be lost on many of them.

There is a family of systems that solves both sets of problems, and that is ranked ballots in multi-member districts: also known as single transferable vote (STV).

The partisan groups (NDP, Green, FVC) would indicate that STV was OK with them, but with their campaign against ranked ballots as well as the way they worded their support it was always considered a "lesser" option.   It is unbelievable that a voting system that would solve a greater number of problems with the current system and be able to create a broad campaign was somehow the "lesser" system.

It was clear to those of us who do not consider party proportionality to be an issue (or for some just not the major issue) that their "we are happy with any system that is proportional" could never be used to create a broad consensus around a version of STV.

STV isn't a single system, but a family of systems that have specific features in common: ranked ballots to solve the vote-splitting and plurality issues, coupled with multi-member districts to solve not only party proportionality but proportionality based on other demographic traits as well.   Our conversation should have been focused around optimizing district magnitude (the rural-urban proposals, and how to group districts), how to allocate surplus votes (random vs deterministic, etc), whether we offer party-promoting features (above the line voting) and whether we solve the "donkey vote" problem (randomizing order of candidates on ballots).

Instead the conversation focused on party-PR, which was a distraction that remained unclear on whether the voting system would improve or get worse.

Back in 2013 there were FVC activists who asked that I apologize for what I wrote in this blog. It turns out that my concern was proven correct, and that their campaign was counterproductive.  I don't expect these advocates to rescind their request for an apology any more than I expect them to apologize for the harm their campaign did to the electoral reform movement in Canada.  Some of these people were people I considered friends, but their devotion to FVC's campaign made it impossible for them to remain friends with me.

So we are where we are.  There is a large rift separating Canadian electoral reformers which might take a long time to heal (if at all). Electoral reform is off the table federally yet again, and the groups who are largely responsible for that failure are meeting together later today to point outward to someone else to blame.