[nznog] Telecom's "peering" change of heart really isn't

[Matthew Poole]

http://computerworld.co.nz/news.nsf/news/EA57668828E0FEEFCC2572AC0070D766

"While emphasing that it's still early days for the proposal and details
need to be worked out, Marshall says providers would pay for circuits into
Telecom's network, and the incumbent would reciprocate for connections
towards providers."

How is this an improvement on the current situation? They're not making
use of existing peering infrastructure, they're just making other
providers pay for a (Telecom-supplied) connection into Telecom's network,
which is an option that's always been available to anyone willing to spend
the money.
Here's hoping that they get told that anything less than true neutral
peering isn't good enough.

[Jonny Martin]

Post by Matthew Poole
"While emphasing that it's still early days for the proposal and details
need to be worked out, Marshall says providers would pay for
circuits into
Telecom's network, and the incumbent would reciprocate for connections
towards providers."

Which is not entirely different to say, buying a 'circuit' so you can
peer across the WIX. Obviously there is a big difference in the
nature of the circuit provider and Telecom though!

Post by Matthew Poole
How is this an improvement on the current situation? They're not making
use of existing peering infrastructure, they're just making other
providers pay for a (Telecom-supplied) connection into Telecom's network,
which is an option that's always been available to anyone willing to spend
the money.

In which case other providers will make their own choice as to
whether or not it makes sense for them to interconnect in this way.

Post by Matthew Poole
Here's hoping that they get told that anything less than true neutral
peering isn't good enough.

That's assuming of course that it is desirable to _mandate_ that all
providers peer. Sounds like a dangerous proposition to me.
Particularly given there are no common and widely accepted
definitions of what peering and other related terms actually mean,
where 'widely' encompasses the whole of industry.

The real worry here is that if Telecom were to end up running with a
faux-peering setup, that they successfully use it as a 'see, we are
good and we peer!' bargaining chip in any regulatory discussion.

All in all I think this is a positive step forward, but let's not
read too much into it just yet.

Cheers,
Jonny.

[Matthew Poole]

Post by Jonny Martin
That's assuming of course that it is desirable to _mandate_ that all
providers peer. Sounds like a dangerous proposition to me.

Why not mandate that any ISP with more than xx,000 users must peer through
the IXs? That's a nice, objective test, and in any case the only ISPs of
significance that would be affected by such an instruction are the ones
owned by TCL and TCNZ. Everyone else already peers.

[Alastair Johnson]

Post by Matthew Poole

Post by Jonny Martin
That's assuming of course that it is desirable to _mandate_ that all
providers peer. Sounds like a dangerous proposition to me.

Why not mandate that any ISP with more than xx,000 users must peer through
the IXs? That's a nice, objective test, and in any case the only ISPs of
significance that would be affected by such an instruction are the ones
owned by TCL and TCNZ. Everyone else already peers.

Every ISP at Every IX? Does it end at ISPs? What about universities,
for instance? What about satellite networks where the uplink isn't
necessarily NZ based?

Are you going to pay for my long haul circuits to Palmy?

Doesn't sound like it would work so well, to me. If we said only the
"major" IXs, what happens when the poor guy in Christchurch signs up his
XX,000th customer and is forced to peer... but isn't getting access to
the Big Boys?

Regulated peering sounds very scary to me...

[Alastair Johnson]

[once more for good luck - maybe THIS time I'll get it on list]

Post by Matthew Poole

Post by Alastair Johnson
Every ISP at Every IX? Does it end at ISPs? What about universities,
for instance? What about satellite networks where the uplink isn't
necessarily NZ based?

At their local (to their primary centre of operations) IX, obviously.
And I'm envisaging xx,000 being a sufficiently high number that only the
University of Auckland (with around 40,000 FTE students) would even come
close.

Okay, but if you're only interconnecting/exchanging at your local IX
then suddenly this is not as useful as it sounded. Telecom's primary
center of operations is indisputably Auckland. Does that mean that,
e.g. Trademe, cannot reach TNZ across an IX?

Or do we come back to region-isation again?

Post by Matthew Poole

Post by Alastair Johnson
Are you going to pay for my long haul circuits to Palmy?

Is your primary centre of operations in Palmy? No? Why would you peer in
Palmy, then?

Because you're regulating peering. If we must peer, then surely
everyone needs to peer either everywhere (expensive), or at one IX
(expensive for some).

Otherwise you're significantly reducing the benefit of it before you
even start.

What happens if my POPs are islands and not connected to my backbone[s]?
(think internap PNAPs) I'm only going to originate local prefixes anyway.

What if I don't want to haul Content Provider C's traffic from Palmy to
my subscribers in Auckland because it's 50+Mbps?

Third option: regional based peering. My POP in PMR peers at the PNIX
and only originates local prefixes.

These all sound like significant business and technical issues that
would need to be addressed by regulation.

Unless your idea of regulation is to just focus on the big boys. Maybe
that's fair, maybe that's not....

Post by Matthew Poole

Post by Alastair Johnson
Doesn't sound like it would work so well, to me. If we said only the
"major" IXs, what happens when the poor guy in Christchurch signs up his
XX,000th customer and is forced to peer... but isn't getting access to
the Big Boys?

Realistically, how many ISPs that are based outside AKL/WLG stand any
chance of getting 50+,000 users? Remembering that of the top five,
unless something has changed a lot in recent times, only Xtra and the
TCL stable are > 100k.

If the government is serious about our OECD rankings, then I think many
ISPs are going to hit >50k. If you believe press releases, Ihug, Orcon,
and Callplus are all around (or over) the 100k mark.

Post by Matthew Poole

Post by Alastair Johnson
Regulated peering sounds very scary to me...

Only if one takes it to absurdity. If it's regulated in such a way that
only the very largest players could be affected, and the requirement on
which IX(s) is based around centre(s) of operations, it's easily
defined, easily enforced, and as I said won't really affect anyone other
than TCL and TCNZ.

Which one may state is not particularly fair.

Incidentally, what about Vodafone? I've noticed that traffic from a
large number of smaller providers currently trombones via Tokyo and Sydney.

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that
most (all?) of them already peer. UoA, which is far-and-away the largest
university in the country, certainly peers at APE.

I think you'll find that UoA in particular does not peer [any longer].
UoW also does not.

aj.

[Simon Lyall]

Post by Alastair Johnson
If the government is serious about our OECD rankings, then I think many
ISPs are going to hit >50k. If you believe press releases, Ihug, Orcon,
and Callplus are all around (or over) the 100k mark.

[Alastair Johnson]

Post by Matthew Poole
How is this an improvement on the current situation? They're not making
use of existing peering infrastructure, they're just making other
providers pay for a (Telecom-supplied) connection into Telecom's network,
which is an option that's always been available to anyone willing to spend
the money.

Says who? Juha, while we all love him, often reports things with an
implied slant which is not always correct.

Private interconnection between carriers has always been done on a cost
incurred by each party basis. ie. We peer in two locations, I pay for
circuit #1, you pay for circuit #2. It sounds like Telecom is offering
something similar here. It may not require to be a Telecom circuit - if
you, say, have DSLAMs located in a Telecom CO for LLU, then surely
you're going to have your own backhaul anyway....

[Juha Saarinen]

Post by Alastair Johnson
Says who? Juha, while we all love him, often reports things with an
implied slant which is not always correct.

You need to take that "implied slant which is not always correct"
complaint to the source of the things reported instead, namely Telecom.

[Don Stokes]

Post by Alastair Johnson
Private interconnection between carriers has always been done on a cost
incurred by each party basis. ie. We peer in two locations, I pay for
circuit #1, you pay for circuit #2. It sounds like Telecom is offering
something similar here. It may not require to be a Telecom circuit - if
you, say, have DSLAMs located in a Telecom CO for LLU, then surely
you're going to have your own backhaul anyway....

But this is sounding rather like, "you, pathetic little ISP, will pay
me, big ugly telco, to reach MY peering point, because your existing
peering arrangements are not worthy of my favour."

That doesn't sound like neutral peering to me. The whole point of
peering is to reduce costs, not just add another way to hand money to
carriers.

A peering point is somewhere I can put a big pat pipe into to exchange
traffic with lots of peers, *including* bilateral arrangements with
transit providers, so I *don't* have to pay for lots of smaller (more
easily congested) pipes to individual carriers, no matter how big and
important they think they are.

Having to be a Telecom customer to play is fine if you're already
wholesaling DSL off them, but I'm getting rather tired of the attitude
that an ISP is someone who wholesales off Telecom that seems to permeate
most of these discussions.

What about the rest of us who are doing stuff that doesn't involve
sponging off Telecom infrastructure?

-- don

[Alastair Johnson]

[it's 4am - I'm bored]

Post by Don Stokes
But this is sounding rather like, "you, pathetic little ISP, will pay
me, big ugly telco, to reach MY peering point, because your existing
peering arrangements are not worthy of my favour."

Perhaps. It's also very common for large carriers (of the telco and
Internet kinds) to interconnect this way and not over public fabric. The
majority of traffic in the US is exchanged via private links; and a
significant amount in Europe as well. A hefty chunk of the traffic LINX
'moves' is private interconnects.

Post by Don Stokes
That doesn't sound like neutral peering to me. The whole point of
peering is to reduce costs, not just add another way to hand money to
carriers.

Interestingly, peering often adds costs or is not that much cheaper than
transit. When transit falls to $10/mbps levels here (heh), you might
find people giving up on peering. The ridiculously cheap transit prices
in the US vs. the cost of building your own infrastructure to IXes has
pushed a number of players to not bother with peering.

Not to say that mass interconnection is bad.

Post by Don Stokes
A peering point is somewhere I can put a big pat pipe into to exchange
traffic with lots of peers, *including* bilateral arrangements with
transit providers, so I *don't* have to pay for lots of smaller (more
easily congested) pipes to individual carriers, no matter how big and
important they think they are.

Nobody's making you pay for circuits to other providers - if you're in a
common location with your own infrastructure, you could interconnect.
If you're not, well, nobody is MAKING you interconnect. You might find
it's cheaper and beneficial for your business to do so... or you might not.

Post by Don Stokes
Having to be a Telecom customer to play is fine if you're already
wholesaling DSL off them, but I'm getting rather tired of the attitude
that an ISP is someone who wholesales off Telecom that seems to permeate
most of these discussions.

Perhaps ISPs should stop focusing on being Telecom (re|whole)salers and
focus on building their own networks and access mechanisms - the
constant PR spiel about how LLU is going to save the world because ISPs
will be putting in 50 million DSLAMs doesn't help here.

Post by Don Stokes
What about the rest of us who are doing stuff that doesn't involve
sponging off Telecom infrastructure?

I think that's awesome and more people should be doing it. Why aren't they?

aj.

[joshua sahala]

Post by Matthew Poole
http://computerworld.co.nz/news.nsf/news/EA57668828E0FEEFCC2572AC0070D766
"While emphasing that it's still early days for the proposal and details
need to be worked out, Marshall says providers would pay for circuits into
Telecom's network, and the incumbent would reciprocate for connections
towards providers."
How is this an improvement on the current situation? They're not making
use of existing peering infrastructure, they're just making other
providers pay for a (Telecom-supplied) connection into Telecom's network,
which is an option that's always been available to anyone willing to spend
the money.
Here's hoping that they get told that anything less than true neutral
peering isn't good enough.

what is your definition of "neutral peering" in this case?

someone always has to pay for the circuit/cross-connect...when Level(3)
and Sprint peer with one another, it isn't via a shared/public VLAN at
Equinix/PAIX/etc, it will be via a private circuit (or cross-connect in an
IX) where they share costs: (3) pays for the first cross-connect, Sprint
the next, rinse, repeat.

settlement-free interconnect is never free (as in beer), there is always
CAPEX/OPEX involved (hardware, MRC, power, NOC, etc).

/joshua

[Matthew Poole]

Post by joshua sahala
what is your definition of "neutral peering" in this case?

"I pay for my circuit to the XYZ IX, you pay for yours, we exchange route
advertisements, traffic follows the shortest route..."
As opposed to "I pay for a circuit to you, you pay for a circuit to me..."

Post by joshua sahala
settlement-free interconnect is never free (as in beer), there is always
CAPEX/OPEX involved (hardware, MRC, power, NOC, etc).

Oh, that much is obvious. But paying for a circuit to a peering point,
maybe with fees charged by the IX to cover the cost of running it (power,
etc), is far different to paying for a circuit into the network of each
and every provider with which you peer.

[Martin Kealey]

[...] there are no common and widely accepted
definitions of what peering and other related terms actually mean,
where 'widely' encompasses the whole of industry.

We need to take off our techie hats and put on our lawyerie or accountant ones.

How about this:

Neutral Peering is peering at a Neutral Location.

A Neutral Location is one where the entity providing the location does not provide connectivity to that location (or does so "at arms length" and competing equally with all other connectivity providers).

A Neutral Location may be any physical size, provided that the location provider does not charge for traffic or bandwidth between participants within the location.

In other words, Citylink metro qualifies as a location encompassing all of Wellington, because there is no charge within the "location". Telecom "peering" does not qualify beccause you can't backhaul your own cable and knock on their door saying "where do I plug in". Or if you could, you wouldn't get peering on an arms-length cost basis.

-Martin

[jamie baddeley]

Post by Martin Kealey

[...] there are no common and widely accepted
definitions of what peering and other related terms actually mean,
where 'widely' encompasses the whole of industry.

We need to take off our techie hats and put on our lawyerie or accountant ones.
Neutral Peering is peering at a Neutral Location.
A Neutral Location is one where the entity providing the location does
not provide connectivity to that location (or does so "at arms length"
and competing equally with all other connectivity providers).

That's a fairly thin argument given the distributed nature of some
exchanges which is enabled by bandwidth.

Post by Martin Kealey
A Neutral Location may be any physical size, provided that the
location provider does not charge for traffic or bandwidth between
participants within the location.
In other words, Citylink metro qualifies as a location encompassing
all of Wellington, because there is no charge within the "location".

The Location is a building inside which the peering fabric resides.
Anything more than that and we're talking about bandwidth. You're
kidding yourself if you think otherwise.

Post by Martin Kealey
Telecom "peering" does not qualify because you can't backhaul your own
cable and knock on their door saying "where do I plug in".

Where does it say that?

It says "Marshall says providers would pay for circuits into Telecom's
network"

it doesn't say

"Marshall says providers would pay for Telecom circuits into Telecom's
network"


---------

Post by Martin Kealey
Okay, but if you're only interconnecting/exchanging at your local IX
then suddenly this is not as useful as it sounded. Telecom's primary
center of operations is indisputably Auckland. Does that mean that,
e.g. Trademe, cannot reach TNZ across an IX?
Or do we come back to region-isation again?

The real issue is to get some consensus on how one defines
regional/local. A sensible starting point is linking that decision to
where the /32's within the networks that are peered are.

Linking it to geography strictly is a great leap *backwards* towards LCA
concepts in my view.

Linking it to concentrations of competitive networks might be sensible.
after all they tend to be where the people are.

The thing to bear in mind that the bigger the metro is the more
expensive it gets and the more it starts to look like national transit.

"Yeah, my metro is 1000KM's long, and I'm going to charge you $160 per
meg to connect with it" - sound familiar?

Post by Martin Kealey

Post by Alastair Johnson
Are you going to pay for my long haul circuits to Palmy?

Is your primary centre of operations in Palmy? No? Why would you peer in
Palmy, then?

Because you're regulating peering. If we must peer, then surely
everyone needs to peer either everywhere (expensive), or at one IX
(expensive for some).

Maybe, but this might be based on the presumption that there's only two
paradigms. Peering, and International. That's clearly wrong. Just like
the view that there's only National and International. That's equally
wrong. The real deal is there's 3 zones. Local, National, and
International. One day there might be regional as well.

The point is that the price is vaguely linked to the cost of providing
the infrastructure to cover the area in question. Of course market
forces will influence that based on the existence of competition. And
competition is related to the relative ease of providing the
infrastructure to cover the area in question in the first place.

What's really at stake here is the industry agreeing to accept that 3
high level grades of (more than 100 metres) transport (in terms of
reach) is a valid concept. Maybe more if we can be sensible and it
doesn't confuse the market.

We used to have 3 grades. Then 2 entities decided there were 2 grades in
2004. Then some got confused that there was only 1 grade. Others got
confused between local and national (or couldn't/wouldn't tell the
difference). But we mostly understood that were there still 3 grades.
Local. National. International.

After all, 3 is the magic number. Ho ho. So we're back at the point
where one of those entities recognises that again. Let's hope that we
can all be sensible. 3-4 years is a while to wait, but I've seen worse.

Post by Martin Kealey
Otherwise you're significantly reducing the benefit of it before you
even start.
What happens if my POPs are islands and not connected to my backbone[s]?
(think internap PNAPs) I'm only going to originate local prefixes anyway.
What if I don't want to haul Content Provider C's traffic from Palmy to
my subscribers in Auckland because it's 50+Mbps?
Third option: regional based peering. My POP in PMR peers at the PNIX
and only originates local prefixes.

And if providers wish to announce more than that, that's at their discretion.

Post by Martin Kealey
These all sound like significant business and technical issues that
would need to be addressed by regulation.
Unless your idea of regulation is to just focus on the big boys. Maybe
that's fair, maybe that's not....

Good point. People! Try to think about living with the consequences after the war. That'll balance what you do today.


Jamie

[joshua sahala]

Post by jamie baddeley

Okay, but if you're only interconnecting/exchanging at your local IX then
suddenly this is not as useful as it sounded. Telecom's primary center
of operations is indisputably Auckland. Does that mean that, e.g.
Trademe, cannot reach TNZ across an IX?
Or do we come back to region-isation again?

The real issue is to get some consensus on how one defines
regional/local. A sensible starting point is linking that decision to
where the /32's within the networks that are peered are.

i'm not sure that i agree that tracking $large_number /32s is sensible
(and i'm pretty sure you aren't talking about ipv6)

Post by jamie baddeley
Linking it to geography strictly is a great leap *backwards* towards LCA
concepts in my view.

this statement contradicts what you said about tracking /32s - a /32 is
tied to a location, either statically or dynamically.

i agree that LCA is a step in the wrong direction

Post by jamie baddeley
Linking it to concentrations of competitive networks might be sensible.
after all they tend to be where the people are.
The thing to bear in mind that the bigger the metro is the more
expensive it gets and the more it starts to look like national transit.

metro != national

no city/metro within new zealand is that large...not even the auckland
sprawl. london, new york, etc. have much larger populations and much
larger areas...but they still are just cities.

Post by jamie baddeley
"Yeah, my metro is 1000KM's long, and I'm going to charge you $160 per meg
to connect with it" - sound familiar?

regardless of how big a network is (or tries to be), is largely
irrelevant. is that network worth $money to you? if it can be had via
another path for $less_money then why bother?

Post by jamie baddeley

Post by Matthew Poole

Post by Alastair Johnson
Are you going to pay for my long haul circuits to Palmy?

Is your primary centre of operations in Palmy? No? Why would you peer
in Palmy, then?

Because you're regulating peering. If we must peer, then surely everyone
needs to peer either everywhere (expensive), or at one IX (expensive for
some).

Maybe, but this might be based on the presumption that there's only two
paradigms. Peering, and International. That's clearly wrong. Just like
the view that there's only National and International. That's equally
wrong. The real deal is there's 3 zones. Local, National, and
International. One day there might be regional as well.

what is local? this sounds a little like LCA again...

france telecom defines "local peering" as:

A local peering refers to interconnections contracted on a
specific country with internet route from this country only.

depending upon whom you ask, you will likely get different definitions for
"local" and "regional"..."national" and "international" are probably the
only terms that will yield a consistent definition

Post by jamie baddeley
The point is that the price is vaguely linked to the cost of providing
the infrastructure to cover the area in question. Of course market forces
will influence that based on the existence of competition. And
competition is related to the relative ease of providing the
infrastructure to cover the area in question in the first place.

very very vaguely related. i haven't seen much 'cost + margin' pricing
since coming to this country...

Post by jamie baddeley
What's really at stake here is the industry agreeing to accept that 3
high level grades of (more than 100 metres) transport (in terms of reach)
is a valid concept. Maybe more if we can be sensible and it doesn't
confuse the market.

are you talking about "transport" (a layer 2 service) or "transit" (a
layer 3 service)?

how much does it really cost to haul one byte of data between any two
points within new zealand? cost + margin

if your network doesn't reach one (or both) of those end points, then you
have two choices, pay someone else to reach them for you, or build your
network to reach them.

Post by jamie baddeley
We used to have 3 grades. Then 2 entities decided there were 2 grades in
2004. Then some got confused that there was only 1 grade. Others got
confused between local and national (or couldn't/wouldn't tell the
difference). But we mostly understood that were there still 3 grades.
Local. National. International.

"local"? "national"? "international"?

sounds like "3 grades" of phone charging...

if you want to purchase 100mbps of ip transit in billings, montana it is
probably going to cost you a bit more per-megabit than 100mbps in tampa,
florida...but there is no differentiation in pricing made between a
destination in kansas or a destination in south africa.

chances are that even if the destination is on different provider in
billings, montana, it will be hauled to chicago and back (~4100km
round-trip) (longer than invercargill to auckland and back - roughly
3000km)

Post by jamie baddeley
After all, 3 is the magic number. Ho ho. So we're back at the point where
one of those entities recognises that again. Let's hope that we can all
be sensible. 3-4 years is a while to wait, but I've seen worse.

i prefer the number 'i' - it seems much more magical than '3'...but that
is just me

Post by jamie baddeley

Otherwise you're significantly reducing the benefit of it before you
even start.

define "benefit"?

to whom?
for which destinations?
from which sources?
for what costs?

Post by jamie baddeley

What happens if my POPs are islands and not connected to my
backbone[s]?
(think internap PNAPs) I'm only going to originate local prefixes anyway.

each pnap was treated as a separate network: each had a unique ASN
(sometimes several), and they tried to purchase transit from the same
providers are each pnap, but it varied a bit. if you were an internap
customer in more than one location with no internal backbone, you had to
play tricks (as-override or similar) or announce inconsistent tables like
internap did...

the last time i was an internap customer, the transit was "cheap" and the
quality was variable. ymmv

fyi - internap has a backbone now, and i think most (maybe all) of the
pnaps are tied together (or will be).

Post by jamie baddeley

What if I don't want to haul Content Provider C's traffic from Palmy to
my subscribers in Auckland because it's 50+Mbps?
Third option: regional based peering. My POP in PMR peers at the PNIX
and only originates local prefixes.

see above re "local"

my rambling two-cents (pick a currency)
/joshua

[jamie baddeley]

Post by joshua sahala

Post by jamie baddeley

Okay, but if you're only interconnecting/exchanging at your local IX then
suddenly this is not as useful as it sounded. Telecom's primary center
of operations is indisputably Auckland. Does that mean that, e.g.
Trademe, cannot reach TNZ across an IX?
Or do we come back to region-isation again?

The real issue is to get some consensus on how one defines
regional/local. A sensible starting point is linking that decision to
where the /32's within the networks that are peered are.

i'm not sure that i agree that tracking $large_number /32s is sensible
(and i'm pretty sure you aren't talking about ipv6)

Not necessarily talking about tracking /32's - more talking about an
approach which recognises where the punters are rather than something
that reflects architecture and therefore where the netblocks aggregate
to. For example one ISP has a number of punters in Palmerston North
which hub off a Chch RAN. If you're not cogniscient of where the punters
are local peering could mean PN punters are announced in Chch instead of
PN. Which makes a mockery of the idea.

Post by joshua sahala

Post by jamie baddeley
Linking it to geography strictly is a great leap *backwards* towards LCA
concepts in my view.

this statement contradicts what you said about tracking /32s - a /32 is
tied to a location, either statically or dynamically.
i agree that LCA is a step in the wrong direction

Post by jamie baddeley
Linking it to concentrations of competitive networks might be sensible.
after all they tend to be where the people are.
The thing to bear in mind that the bigger the metro is the more
expensive it gets and the more it starts to look like national transit.

metro != national

Yes. My tongue was in my cheek then.

Post by joshua sahala
no city/metro within new zealand is that large...not even the auckland
sprawl. london, new york, etc. have much larger populations and much
larger areas...but they still are just cities.

Post by jamie baddeley
"Yeah, my metro is 1000KM's long, and I'm going to charge you $160 per meg
to connect with it" - sound familiar?

regardless of how big a network is (or tries to be), is largely
irrelevant. is that network worth $money to you? if it can be had via
another path for $less_money then why bother?

The point of that comment is the lack of ability to distinguish between
national and local/metro.

Post by joshua sahala

Post by jamie baddeley

Post by Matthew Poole

Post by Alastair Johnson
Are you going to pay for my long haul circuits to Palmy?

Is your primary centre of operations in Palmy? No? Why would you peer
in Palmy, then?

Because you're regulating peering. If we must peer, then surely everyone
needs to peer either everywhere (expensive), or at one IX (expensive for
some).

Maybe, but this might be based on the presumption that there's only two
paradigms. Peering, and International. That's clearly wrong. Just like
the view that there's only National and International. That's equally
wrong. The real deal is there's 3 zones. Local, National, and
International. One day there might be regional as well.

what is local? this sounds a little like LCA again...
A local peering refers to interconnections contracted on a
specific country with internet route from this country only.
depending upon whom you ask, you will likely get different definitions for
"local" and "regional"..."national" and "international" are probably the
only terms that will yield a consistent definition

I'm asking NZNOG. France Telecom would say that in the context you
describe which is international. Europe is bound to have quite a
different perspective due to the landlocked nature of many of the
countries. Unlike NZ.

Post by joshua sahala

Post by jamie baddeley
The point is that the price is vaguely linked to the cost of providing
the infrastructure to cover the area in question. Of course market forces
will influence that based on the existence of competition. And
competition is related to the relative ease of providing the
infrastructure to cover the area in question in the first place.

very very vaguely related. i haven't seen much 'cost + margin' pricing
since coming to this country...

Correlating cost against market price assists you in the context of
assessing price-gouging/monopoly behaviours.

Post by joshua sahala

Post by jamie baddeley
What's really at stake here is the industry agreeing to accept that 3
high level grades of (more than 100 metres) transport (in terms of reach)
is a valid concept. Maybe more if we can be sensible and it doesn't
confuse the market.

are you talking about "transport" (a layer 2 service) or "transit" (a
layer 3 service)?

Either. I am sure that parties will develop products that are
characterised as one or the other.

Post by joshua sahala
how much does it really cost to haul one byte of data between any two
points within new zealand? cost + margin
if your network doesn't reach one (or both) of those end points, then you
have two choices, pay someone else to reach them for you, or build your
network to reach them.

Sure.

Post by joshua sahala

Post by jamie baddeley
We used to have 3 grades. Then 2 entities decided there were 2 grades in
2004. Then some got confused that there was only 1 grade. Others got
confused between local and national (or couldn't/wouldn't tell the
difference). But we mostly understood that were there still 3 grades.
Local. National. International.

"local"? "national"? "international"?
sounds like "3 grades" of phone charging...

Well it's how it was here prior to 2004. And it seemed to work.

Post by joshua sahala
if you want to purchase 100mbps of ip transit in billings, montana it is
probably going to cost you a bit more per-megabit than 100mbps in tampa,
florida...but there is no differentiation in pricing made between a
destination in kansas or a destination in south africa.

I suspect that's related to the retail side rather than the wholesale
side.

Post by joshua sahala
chances are that even if the destination is on different provider in
billings, montana, it will be hauled to chicago and back (~4100km
round-trip) (longer than invercargill to auckland and back - roughly
3000km)

Personally I think it's more fruitful for us to work out what works for
us rather than trying to compare to markets that have a significantly
different dynamic and economic equation. The US is nationally
significantly larger, and pays significantly less to reach the world.

Post by joshua sahala

Post by jamie baddeley

What happens if my POPs are islands and not connected to my
backbone[s]?
(think internap PNAPs) I'm only going to originate local prefixes anyway.

each pnap was treated as a separate network: each had a unique ASN
(sometimes several), and they tried to purchase transit from the same
providers are each pnap, but it varied a bit. if you were an internap
customer in more than one location with no internal backbone, you had to
play tricks (as-override or similar) or announce inconsistent tables like
internap did...
the last time i was an internap customer, the transit was "cheap" and the
quality was variable. ymmv
fyi - internap has a backbone now, and i think most (maybe all) of the
pnaps are tied together (or will be).

Sounds familiar. I'd suggest that that evolutionary path is one that is
also tread here.

[Matthew Poole]

Post by Alastair Johnson
Every ISP at Every IX? Does it end at ISPs? What about universities,
for instance? What about satellite networks where the uplink isn't
necessarily NZ based?

At their local (to their primary centre of operations) IX, obviously. And I'm
envisaging xx,000 being a sufficiently high number that only the University of
Auckland (with around 40,000 FTE students) would even come close.

Post by Alastair Johnson
Are you going to pay for my long haul circuits to Palmy?

Is your primary centre of operations in Palmy? No? Why would you peer in Palmy,
then?

Post by Alastair Johnson
Doesn't sound like it would work so well, to me. If we said only the
"major" IXs, what happens when the poor guy in Christchurch signs up his
XX,000th customer and is forced to peer... but isn't getting access to
the Big Boys?

Realistically, how many ISPs that are based outside AKL/WLG stand any chance of
getting 50+,000 users? Remembering that of the top five, unless something has
changed a lot in recent times, only Xtra and the TCL stable are > 100k.

Post by Alastair Johnson
Regulated peering sounds very scary to me...

Only if one takes it to absurdity. If it's regulated in such a way that only
the very largest players could be affected, and the requirement on which IX(s)
is based around centre(s) of operations, it's easily defined, easily enforced,
and as I said won't really affect anyone other than TCL and TCNZ.

Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the largest university
in the country, certainly peers at APE.

[Hamish MacEwan]

Post by Matthew Poole
Here's hoping that they get told that anything less than true neutral
peering isn't good enough.

The question is what does the telling, mandate by the state is how
carriers have been regulated for ever, and not alltogether
successfully... The hope has been that competition would allow the
market (mainly the customers) inform behaviour.

Unfortunately, the customer suffers from the "seen/unseen" dichotomy
and they don't see peering as a significant issue, what they see
works, even badly, and they're mostly happy with that.

We elect individuals to specialise in this area on our behalf and it
was sad to see David Cunliffe parroting the Telecom line about
"peering is complex." Umm, if it was, would so many small players
and, lo!, even customers be doing it? Lets separate peering from
Telecom and say the former is simple, the latter complex.

As for regulation, there is the Commerce Act, which talks about the
use of market power to suppress competition. Well, the only people
who don't peer (in the APE/WIX bilateral permissionless way) are,
oddly, large incumbents. Could be coincidence, but comments like
Don's suggest there is an issue of market power that can lead to
demands others cannot make. Not that this is something ComCom has
ever AFAIK investigated, perhaps its unseen by them as an issue.

The "Bill & Keep" model, where you charge your customers and don't
attempt to indirectly charge the customers of other suppliers, is
achieving more traction. What ever costs autonomous operators incur,
they charge their customers. So if your customers are pulling/pushing
a lot of traffic from somewhere/someone else, minimise the costs or
charge your customers. The idea that someone else is responsible for
the costs your customers incur is the wildest economic externality
I've ever seen swallowed, TCO excepted of course.

Matthew Bolland's comments from the cover of Telecommunications Review
are at best ironic, "If ISPs choose to use overseas connections for
national transit when, for half the price they can buy our national
service, then that is their choice."

And its a choice that TCL makes when it prefers to take RNZ from their
US server than take it locally... I guess choosing the higher cost
option is only stupid when you're smaller than them.

Post by Matthew Poole
Matthew Poole

Hamish.

[Stewart Fleming]

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the largest university
in the country, certainly peers at APE.

Help me out here. What incentives can I (as operator of a local
exchange) now offer to a University to convince it that peering locally
is a Good Idea when they have KAREN? At various times, I've offered the
University Of Eastern Venezuela (*) peering for a) $XXX/monthly (**),
and b) the cost of a piece of patch cord, so far without takers. I
think I might have to pay them before the chicken/egg dichotomy gets
resolved ("We'll peer if everyone else does"; Everyone Else says "We'll
peer if they do".)...

(*) Name changed to protect the innocent.
(**) Amount obscured, but the number of digits is right.

[Gerard Creamer]

Post by Stewart Fleming

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the largest university
in the country, certainly peers at APE.

Help me out here. What incentives can I (as operator of a local
exchange) now offer to a University to convince it that peering locally
is a Good Idea when they have KAREN?

It's the community sharing thing - get the big local content generators
and consumers on there (City Council, University, hospital, local ISPs
etc) and it should all work out. Maybe do the big 3 free for the first
X months and then they can see if its worth renewing / subscribing - if
it's not saving them more than it'd cost to pass it through some
pay-per-byte provider then peering isn't viable in your community.
Economics can be cruel.

Regards,
Gerard

[Glen Eustace]

Post by Stewart Fleming

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the largest university
in the country, certainly peers at APE.

Help me out here. What incentives can I (as operator of a local
exchange) now offer to a University to convince it that peering locally
is a Good Idea when they have KAREN? At various times, I've offered the
University Of Eastern Venezuela (*) peering for a) $XXX/monthly (**),
and b) the cost of a piece of patch cord, so far without takers. I
think I might have to pay them before the chicken/egg dichotomy gets
resolved ("We'll peer if everyone else does"; Everyone Else says "We'll
peer if they do".)...

Massey University are peering at WIX and PNIX, not APE as the cost to
get across from the North Shore was more than we were willing to pay.
Other than Inspire, I think we are still the only player on PNIX but
hoped that by being there, we might encourage others in PN to peer also.

In the case of WIX, there are lots of other parties in Wellington that
peer and in the spirit of providing a better service between them and us
than their Internet connection which may well be small, we felt it was
an appropriate environment to support.

[Simon Lyall]

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the largest university
in the country, certainly peers at APE.

Do Auckland University actually Peer? I can't seem to find a route that
goes to them across APE (all traceroutes go via TelstraClear) and their
session with the route servers appears to be down

[Simon Lyall]

Post by Simon Lyall
Do Auckland University actually Peer? I can't seem to find a route that
goes to them across APE (all traceroutes go via TelstraClear) and their
session with the route servers appears to be down

A couple of people have emailed me offlist and said that Auckland Uni
depeered a couple of months ago and is now using Telstra for all traffic
(except KAREN I guess). I'm not sure why this change in policy occured.

[Nathan Ward]

Post by Stewart Fleming

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the
largest university
in the country, certainly peers at APE.

Help me out here. What incentives can I (as operator of a local
exchange) now offer to a University to convince it that peering locally
is a Good Idea when they have KAREN? At various times, I've
offered the
University Of Eastern Venezuela (*) peering for a) $XXX/monthly (**),
and b) the cost of a piece of patch cord, so far without takers. I
think I might have to pay them before the chicken/egg dichotomy gets
resolved ("We'll peer if everyone else does"; Everyone Else says "We'll
peer if they do".)...
(*) Name changed to protect the innocent.
(**) Amount obscured, but the number of digits is right.

They probably don't see a need for it, economically.

Perhaps if you provide them some idea of expected traffic volumes,
and expected performance gains (ie. latency), in a numerical form,
they can translate that in to saved $ per month. Perhaps show how the
growth is going currently, so they can see that it becomes a better
idea over time - that way if they don't do it now, they won't have
ruled it out for the future.

How does KAREN impact this price wise? Can they use KAREN to get to
Auckland, to then get domestic transit from $NSP?

Do they understand that you're simply trying to get higher
performance to/from their users/servers, as opposed to access to the
KAREN network?

--
Nathan Ward

[Ian Batterbee]

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the largest university
in the country, certainly peers at APE.

AUT doesn't peer with APE because it seems a little redundant as we have
a connection through Orcon, who peer at APE

We also connect to TCL and to KAREN.

[Alastair Johnson]

Post by Ian Batterbee

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the largest university
in the country, certainly peers at APE.

AUT doesn't peer with APE because it seems a little redundant as we have
a connection through Orcon, who peer at APE

You might want to ask Orcon to announce your /16 to APE peers then. A
quick check from a couple of APE connected hosts shows traffic hitting
TelstraClear, then AUT.

[Nathan Ward]

Post by Alastair Johnson

Post by Ian Batterbee

Post by Matthew Poole
Oh, and on your point about universities, I think you will find that most
(all?) of them already peer. UoA, which is far-and-away the
largest university
in the country, certainly peers at APE.

AUT doesn't peer with APE because it seems a little redundant as we have
a connection through Orcon, who peer at APE

You might want to ask Orcon to announce your /16 to APE peers then. A
quick check from a couple of APE connected hosts shows traffic hitting
TelstraClear, then AUT.

Assuming these providers also buy domestic transit from TelstraClear,
AS_PATHs look like:
- TelstraClear AUT
- APE Orcon AUT
Shortest AS_PATH wins.

If they buy it from some other transit provider, who interconnects
with TelstraClear:
- <transit> TelstraClear AUT
- APE Orcon AUT
Assuming MED, etc. are the same, the oldest path will be selected. As
APE routers get poked from time to time to add prefixes, the
<transit> provider is likely to be the more stable.


So overall, based on a few assumptions, TelstraClear wins.
It's at this point that I encourage APE/WIX(1) peers to make sure you
prefer routes learned over APE/WIX. LOCAL_PREF works well here to
avoid AS_PATH length and age issues.

--
Nathan Ward


(1) And PNIX and SIX and CHIX and .. you get the idea.

[Alastair Johnson]

Post by Nathan Ward
Assuming these providers also buy domestic transit from TelstraClear,
- TelstraClear AUT
- APE Orcon AUT
Shortest AS_PATH wins.
If they buy it from some other transit provider, who interconnects
- <transit> TelstraClear AUT
- APE Orcon AUT
Assuming MED, etc. are the same, the oldest path will be selected. As
APE routers get poked from time to time to add prefixes, the
<transit> provider is likely to be the more stable.

Oldest path is configurable behavior - many people prefer the more
deterministic lowest neighbor ID because it prevents neighbor triggered
traffic switches.

You're also assuming that all providers use the Route Servers and not
bilateral peering; which is a risky assumption.

Post by Nathan Ward
So overall, based on a few assumptions, TelstraClear wins.
It's at this point that I encourage APE/WIX(1) peers to make sure you
prefer routes learned over APE/WIX. LOCAL_PREF works well here to
avoid AS_PATH length and age issues.

I'd generally assume anyone has a local_pref based on cost: ie:

highest local_pref for customer (ie. "getting paid for") prefixes
second highest for second lowest cost/efficiency (private peering)
third highest for third lowest (public)
fourth highest for transit (or other)

Which fixes any problems with AS_PATH length.

You raise some valid points... except the prefix in question wasn't
announced by any peer that I could see in any case, so there was no
determination being made.

[Nathan Ward]

Post by Alastair Johnson
Oldest path is configurable behavior - many people prefer the more
deterministic lowest neighbor ID because it prevents neighbor
triggered
traffic switches.

But it's not disabled default. My bet is that most people don't
really consider the pros/cons of each in great detail, and instead
opt for just-make-it-go.

Post by Alastair Johnson
You're also assuming that all providers use the Route Servers and not
bilateral peering; which is a risky assumption.

[Alastair Johnson]

Post by Nathan Ward

Post by Alastair Johnson
Oldest path is configurable behavior - many people prefer the more
deterministic lowest neighbor ID because it prevents neighbor
triggered
traffic switches.

But it's not disabled default. My bet is that most people don't
really consider the pros/cons of each in great detail, and instead
opt for just-make-it-go.

Perhaps. However the 'default' you refer to is also not common to all
vendors, however.

I would be interested to know how many people (ISPs) do elect to use
"bgp bestpath compare-routerid" on their Cisco equipment. I'd imagine
smaller/less experienced ASNs probably do leave it, but I suspect one or
two ISPs may well have changed it.

Post by Nathan Ward

Post by Alastair Johnson
You're also assuming that all providers use the Route Servers and not
bilateral peering; which is a risky assumption.

[Joe Abley]

Post by Nathan Ward
Assuming these providers also buy domestic transit from TelstraClear,
- TelstraClear AUT
- APE Orcon AUT
Shortest AS_PATH wins.

I'd presume that highest LOCAL_PREF wins, actually.

If I was running an ISP, I'd probably ensure that I had an import
policy which promoted private peering > public peering > paid peering

Post by Nathan Ward
transit in general.

Anybody who doesn't incorporate such policy into their network
presumably doesn't care how much it costs (or how long it takes) to
deliver a packet, and hence it doesn't matter which way the packets
go. Assuming that any of this matters, customers of such an ISP will
presumably migrate elsewhere naturally, and the problem (from the
perspective of the AS originating the routes) will solve itself.


Joe

[Dean Pemberton]

Post by Joe Abley
Anybody who doesn't incorporate such policy into their network
presumably doesn't care how much it costs (or how long it takes) to
deliver a packet, and hence it doesn't matter which way the packets
go. Assuming that any of this matters, customers of such an ISP will
presumably migrate elsewhere naturally, and the problem (from the
perspective of the AS originating the routes) will solve itself.

What has been lacking thus far is any objective measurement of customer
experience across different ISPs.
Customers are not keen to move ISPs, change email addresses, change bank
APs etc if they are not assured of a better service.

If ISPs would adopt some neutral measurement capture, then there would
be a way to say "Blah ISP is better because they <do thing X>, and Blerg
ISP is worse because they don't"

Some of the work Nevil Brownlee presented at the conference could feed
into this. So could the NLANR AMP stuff (now many ISPs have this
deployed again?). Even the nzdsl speed test stuff.

I don't know if there is a speedtest server on the WIX/APE, but if there
were then you'd be able to get a good appreciation of how the different
ISPs fared when it came to a similar service.


Dean

[joshua sahala]

Post by Dean Pemberton

Post by Joe Abley
Anybody who doesn't incorporate such policy into their network
presumably doesn't care how much it costs (or how long it takes) to
deliver a packet, and hence it doesn't matter which way the packets
go. Assuming that any of this matters, customers of such an ISP will
presumably migrate elsewhere naturally, and the problem (from the
perspective of the AS originating the routes) will solve itself.

What has been lacking thus far is any objective measurement of customer
experience across different ISPs.

the very nature of of the phrase "customer experience" is subjective...

Post by Dean Pemberton
Customers are not keen to move ISPs, change email addresses, change bank
APs etc if they are not assured of a better service.
If ISPs would adopt some neutral measurement capture, then there would
be a way to say "Blah ISP is better because they <do thing X>, and Blerg
ISP is worse because they don't"

define "better" - to which destinations from which sources and for what cost?

Post by Dean Pemberton
Some of the work Nevil Brownlee presented at the conference could feed
into this. So could the NLANR AMP stuff (now many ISPs have this
deployed again?). Even the nzdsl speed test stuff.
I don't know if there is a speedtest server on the WIX/APE, but if there
were then you'd be able to get a good appreciation of how the different
ISPs fared when it came to a similar service.

a millisecond or two? a kilobit or three? are you testing with an o/s
which has had network setting optimized? does the average user have those
same optimized settings?

would 90% of dsl users inside nz even notice the difference?

if 95%+ of the content that people want is hosted offshore, you can peer
at two or twenty-nine or 300 locations within new zealand and it isn't
going to change the "customer experience". in fact, it would likely
decrease performance as the overhead of maintaining more and more hardware
and infrastructure increases and traffic engineering is significantly
complicated.

peering can be a good thing, but it isn't the panacea that it is often
sold as within new zealand.

/joshua

[Dean Pemberton]

Joshua,

Customer experience is very subjective - but tends to be the basis by
which companies try to attract new customers, and by which customers
choose to leave their current provider. So objective, subjective or
whatever, it needs to be addressed.

Similar with 'Better'. You seem to be attacking the fact that these
things are not nailed down to "Latency to site X is less than Y ms" or
"There is under Z% packet loss to site X", when in fact thats not what
the majority of customers care about anyway.

I agree that 'peering' (whatever that means) isn't a silver bullet to
any particular problem. I would suggest that we work harder to define
the perceived problem, which I suspect is defined somewhere in the
middle of:

"X% packet loss and RTT of Yms to site Z"
and
"My Internet seems really slow today"


Dean

Post by joshua sahala

Post by Dean Pemberton

Post by Joe Abley
Anybody who doesn't incorporate such policy into their network
presumably doesn't care how much it costs (or how long it takes) to
deliver a packet, and hence it doesn't matter which way the packets
go. Assuming that any of this matters, customers of such an ISP will
presumably migrate elsewhere naturally, and the problem (from the
perspective of the AS originating the routes) will solve itself.

What has been lacking thus far is any objective measurement of customer
experience across different ISPs.

the very nature of of the phrase "customer experience" is subjective...

Post by Dean Pemberton
Customers are not keen to move ISPs, change email addresses, change bank
APs etc if they are not assured of a better service.
If ISPs would adopt some neutral measurement capture, then there would
be a way to say "Blah ISP is better because they <do thing X>, and Blerg
ISP is worse because they don't"

define "better" - to which destinations from which sources and for what cost?

Post by Dean Pemberton
Some of the work Nevil Brownlee presented at the conference could feed
into this. So could the NLANR AMP stuff (now many ISPs have this
deployed again?). Even the nzdsl speed test stuff.
I don't know if there is a speedtest server on the WIX/APE, but if there
were then you'd be able to get a good appreciation of how the different
ISPs fared when it came to a similar service.

a millisecond or two? a kilobit or three? are you testing with an o/s
which has had network setting optimized? does the average user have those
same optimized settings?
would 90% of dsl users inside nz even notice the difference?
if 95%+ of the content that people want is hosted offshore, you can peer
at two or twenty-nine or 300 locations within new zealand and it isn't
going to change the "customer experience". in fact, it would likely
decrease performance as the overhead of maintaining more and more hardware
and infrastructure increases and traffic engineering is significantly
complicated.
peering can be a good thing, but it isn't the panacea that it is often
sold as within new zealand.
/joshua

[Craig Whitmore]

Post by Dean Pemberton
Some of the work Nevil Brownlee presented at the conference could feed
into this. So could the NLANR AMP stuff (now many ISPs have this
deployed again?). Even the nzdsl speed test stuff.
I don't know if there is a speedtest server on the WIX/APE, but if there
were then you'd be able to get a good appreciation of how the different
ISPs fared when it came to a similar service.

I've over 2 million tests done from NZ DSL connections (For National
Speeds), and overall there is no real difference on average in speeds per
ISP (Nationally)(For DSL users) (even considering where the speedtest
server is @ Orcon).

Regarding the Speedtest.net results showing Airnet #1 and ICONZ #2, as the
overall stats are taken from every test done (even on faster connections),
So they will be skewed on ISP's who have more tests done on Higher Speed
Connections (Wireless,Fibre etc).

International Speed for DSL users per ISP is a different story, but I don't
have enough results for any conclusion.

Thanks