IPv6 Working Group session
RIPE 86
25 May 2023
11 a.m.
Ngi
JEN LINKOVA: Welcome. Please take your seat. We have a packed agenda so it would be great to start right on time. Please turn your mobile phone to like silent mode. The exhibit is over there, the flight attendant will guide you if needed.
This is IPv6 Working Group. Welcome. I am Jen, my co‑chairs is Raymond and Benedikt hiding in the first row. So, who has read meeting minutes from last time? Thank you very much. Do you approve them? Are you happy? Okay, sold. Approved! Please read them actually if you said anything last time. Don't complain later. We are going to publish them and declare them approved.
Okay, we'll have some announcements at the end of the session. If you go into the microphone, please say who you are. It actually does help with meeting minutes. And actually, you see it's not going to make sense, it's not my slide. I think the agenda is actually outdated. We have one quick lightning talk at the beginning of the session.
SPEAKER: Thank you very much. I will be here for just two minutes. Actually, I am tie fun and I am from the RIPE NCC learning and development team. So I am here for this QR code. I will be just pointer for this. So, we are trying to a new advanced IPv6 course in our learning and development team. It will be an e‑learning course actually, and it will be published on our website academy [at] ripe [dot] net and everybody can reach is freely. It go a modular one and it will be including some lab activities as well. So, for this course, we need your input, and we have a questionnaire in this link here, so we have the QR code as well so you can check it. I promise that it will take no more than three or five minutes at most. So, that's it. Thank you very much.
JEN LINKOVA: Thank you very much day fun for doing this. People keep complaining that IPv6 is very hard and complicated:
.
So, what do we have on the agenda next? The floor is yours.
RINSE KLOEK: Thank you. I would like to encourage you to look at the RIPE NCC courses. I have done the IPv6 basic and fundamentals and security courses and encourage you they are very good material to start doing v6.
Deploying v6. Some background about this presentation. I'm working for an ISP and that ISP is growing and it wants to grow than more to 1 million end users mainly consumer connections. But the issue is they don't have a large block of v4 space so they run out of v4 and they had to by v4 for the last two years, and as the price of v4 is increasing, this is the last /16 I bought, this is the last PO I sign. Please seek an alternative because it's hurting or business case. So there was a start of a new project. The IPv6 project. But next to deploying v6 only, it's not achievable right now so we also had to choose for a transition technique, so a new project was born, IPv6 end of transition technique (and a)
.
Transition techniques.
We did some research, we asked Sander Steffann to do some research, he gave us a tip please look at MAP‑T and map E, we did some research but we also asked our CPE vendor could you support one of these proposals, but they said no we won't support it and we are also (protocol) planning to do it so that one fell off.
The next one is NAT64, also a nice translation technique, it's stateful but the issue is we don't have IPv6 role‑out right now and it would like a lot of time to finish the project after implementing NAT64. And also we had some legacy part of the network that was out of support where we don't want to change the configuration any more. So v6 implementing on out of hardware we didn't like to do that.
Then last but not least, NAT 444, also known as CG NAT. One the pros is is works over your existing v4 network, you don't need to change its configurations of your CPEs, it just can work. So we choose to do it next to our v6. The statement of the CFO was I would like to save on v4 cost as soon as possible, so we choose to start implementing Carrier‑Grade NAT before doing v6, wee as engineers didn't like that very much because if we thought if you do CGNAT you have also to do v6. Like the business wants to have Carrier‑Grade NAT first so we started implementing CGNAT.
One of the reasons, there is a good business case for CGNAT. Run the reasons is you see a graph on the right on the X axis you see the big usage per user, per customer and on the Y axis you see the cost of CGNAT per user and if you do an average of five megabits per you're I think that's more or less an average of consumer ISP, the cost of CGNAT will be around €50. I made some basic calculations, it's not dataseted but I took an account of that we have two end solutions so two different NAT boxes to good redundancy. And if you compare those €50 per user with the current price of v4, that's peeked to about 40 to €50 thanks year, I think there is a good business case.
So, a company decides implement ‑‑ from public v4 so C G grade. Like the prices are still very high, they are a bit decreasing but it's only for the smaller blocks, if you have the small v4 blocks, like the /16, you still get a decent price, it's a business case.
And also do it as soon as possible. Prices are decreasing.
One of the things that can kill your business case, it's the support call. Support calls cost a lot of money so you don't want users F you migrating from public to CGNAT to call our service desk, so we did a couple of things. First of all, we only migrated user that didn't user surges like port mapping and ‑‑ we had some ACS system to see what kind of users had port forwarding for example for the home surges and we decided don't migrate otherwise they will complain and will call and will kill your business case.
Another thing is we also made the option that the users can opt out from CGNAT. They could log into portal on the app and still I still want my public v4 just by clicking on a button.
Last but not least, testing is very important. We had our own lab at the company with all kinds of equipment, PlayStation 4, 5, tested all kinds of games, so we had an employee that was allowed to test all day with gaming. So it was a pretty nice job for him. And we didn't see any big issues. Like every application that worked out of the box and next to that I decided to be the first test user, so I said just said migrate my home connection, and starting from February to now, I am fully in Carrier‑Grade NAT without v6 but I didn't have any issues. The only issue I had was connecting to my home assistant at home because I didn't have for the forwarding but that was expected.
The next thing is if you implement is, to improve the basis case you can do a couple of things. The first thing we did is bypass the internal traffic to two carrier NATs. But bypass it from the CGNAT boxes, for example mill DNS, I think it's very important to lower a number of sessions and the load on those boxes. Another nice thing is Google had its, I think a couple of years yet is that you can, if you have those nice caching boxes from Google you can ask them, I'll use communities to announce those Carrier‑Grade DPS to those boxes then the traffic from the Google caches will directly be served without going to the CGNAT box and it will save you a lot of traffic. Also NetFlix does it, starting from previous month, they have also an option that you directly announce those CGNAT ranges to the caching boxes and the traffic will be served directly. So those two are the big traffic it's nice you can bypass it from your CGNAT box.
Implementing v6, that I think is one of the most important things. After we implement CGNAT we saw if we implement v6, we can dramatically decrease the traffic to our CGNAT boxes. So finally, we have a business case for IPv6. So implementing v6 will decrease your load and it will decrease the cost per user.
Now about v6. Like we're on the v6 Working Group so I won't talk about CGNAT any more. You see the high level design of the network of the ISP. I'm doing a project. The implementation of v6, it was very, very good to do. It was not very hard. Mostly on the core side. The core side, BGP peering, all the access boxes, it worked out of the box, no any big issues. Most issues we have on the edges of our network. Like the CPE side, a lot of CPE vendors are still very v4 focussed so v6 is pretty new for them so you have to ask them to fix things, to change things. Also some bug fixes like firewalling that was not good implementing, and also for example for v4 you have the option to do port forwarding, but if you have v6 and you want to, for example, open a port for a servers behind a he CPE, that option was not in the CPE, so you could only turn on the firewall but not change some entries in the firewall. That's one thing the vendors are no so focused. You have to ask them to implement v6.
The other thing on the IT and services side, we have some processes like legal logging for example to log what user has what v6 block. It's also very v4 focused so old scripts and processes in the IT departments that fully adapted to v4 but not adapted to v6. So that still needs sometime and we are now finishing that last part of the implementation.
Something more about the implementation. The starting document I used for the implementation was the RIPE 609ment to. I think it's a very good starting point. It has some (690. ? (About prefix assignment. It made my life easier. We chose to do a/56 because we didn't have a very large v6 block from RIPE. I discussed them with them about it but that's another topic.
And also we chose to do, give customers as well as IANA prefix a/1281 prefix for the one link and also prefix delegation a/56. One of the things we chose for that separate/128 that it's backwards compatible for compliance that don't do prefix delegation. For example if you connect your laptop that doesn't do T you can still access the Internet via v6.
We also chose to do persistent IPv6 prefix. That's one the topics in the RIPE document, based on the DHCP option 37, that's a unique connection ID that's put on the link to the customer so we don't use a MAC address or a dual ID but we use that option, and one of the reasons is that for example if the customer swabs a CPE, then it will still get the same prefix. So you can just swap out. It doesn't matter what kind of device, you will always get the same prefix.
One of the reasons we are doing persistence prefixes is if you rotate the prefixes you can get some issues like not every (persist) device in your network or firewall can an adapt to the whole prefix shopping process so to overcome this whole issue and don't go into a rabbit hole we choose to to persist prefixes. I think it's best practice for end users.
And the next thing is also we chose to drop the DHCP v6 release messages. Like if you reboot a CPE. Some much them do the v6 release and the CPE thinks release I delete that had entry from the release table, after the reboot you can get a new prefix. So we chose to just drop that release at the network side so you will always keep your leases stick.
Another challenge we had was the hardware resources. We used some aggregation router. Aggregation how are has limited hardware resources nor the Arp and enabled discovered entries. First of all, we implemented with as well as the single one address as the prefix delegation, and if you do that, it will consume two neighbour discovery entries in your router and as v6 entries take more hardware entries than v4, you can see in the table at the top right, that at the end you had 8 plus 6 is about 14 hardware entries if you do dual stack. So implementing v6 can hurt your business case or your, the hardware because we use that, those Broadcom based boxes they have very limited resources. After we saw this happening we chose to change the configuration of the CPE to only do the v6 prefix delegation and don't use that one prefix any more. It's still supported in the network for old devices but we preferred not to do that. It can save us some hardware space.
I see I'm almost running out of time so I'll skip some things.
Something I'd like to mention is one thing is about the upgrade of the hardware. We saw that the DHCP releases were not sent after ‑‑ were sent after are reboot so we chose to firewall that.
And also one of the things I would like to mention is that the last thing is that we had some testing, it's nor about CGNAT. The only thing that did not work was some from Smart TVs, sometimes they have some options you can start and for example the MPO app you can watch TV shows back, but the customers were complaining everything works except if we watch television via the something, what was the cases in we discovered that CGNAT box does DNS interception by default. You don't see it in the configure, it's just there, it's working. And it doesn't do anything. It just intercepts but doesn't mangle the packets. It does one thing it limits the DNS rate. So like the MPO update a lot of DNS requests it did a burst and that burst didn't arrive and that was the reason it didn't work. So, we asked it to CGNAT box vendors, you have to disable this feature because it's on by default and after disabling it worked perfectly.
Conclusions. There is a small business case for IPv6 if you deploy CGNAT, you will want to have a less traffic as possible through your CGNAT boxes so I think if you start doing it, v6 is a business case like if we have low traffic, you don't need to invest a lot of money in buying new CGNAT boxes. Give the users to do an option to opt out from Carrier‑Grade NAT because you cannot have users that have some legacy application that doesn't work so you can just say just opt out you get a public v4 address.
We only had a couple of users doing that so after migration, we excluded the user that the port forwarding and by doing that, only a handful of users called and go for the opt out option though, so most of the users didn't complain.
Another thing I want to point out is most of the challenges are the edges so all core network something working fine, only the CPE and the IT processors were sometimes a problem.
Persist prefix it is strongly recommended. Be aware of the growth of your neighbour's configure table.
They said "we don't just enable IPv6 because of the business case, it was also because of our customers want it." It's not a business case option only. It's also our competitors in the Netherlands do v6 so we have to do it because it's the only way forward.
Okay, thank you for your attention to my presentation. So if you have any questions, walk to the mic.
JEN LINKOVA: Yes. Thank you very much. We have two questions online, both from Janus Nickalopalus.
First is CPE vendors are paid to develop features. We won't support MAP‑Ts just to reason to change CPE vendor. By the way quite a few CPEs support map AA MAP‑T, I can name at least four, who is your vendor and what is the real story behind the denial?
RINSE KLOEK: We have a vendor that was selected two years ago, we have some relationship. It is Nokia, so I can just mention the name. And it was also pretty confused. Like I mention about those techniques and they told us no, that's not the way to go. So I tried to convince them, convince them, but it was pretty hard to conintense them that these transition techniques are good options. So some way or another I stopped fighting that battle, but I agree, I think that the CPE vendors can do a very good job. We also had a lot of small bugs with CPE vendor, so we also thought if they also to implement another new technique, maybe that will give some more issues. So I agree, I think they should do it, but after I had a battle for a long time I stopped and I think okay, right.
JEN LINKOVA: And the second question is" it was not very clear to me. I had deployed a NAT 444 with native v6 support for example a v4 and v6 in different VRFs and the end user gets both."
RINSE KLOEK: How we are deploying it currently is we are deploying CGNAT so the user is getting an internal IP address, a 164 address and what we are planning is that the customers also get a v6 prefix, so at the end, the user will have dual stack. So now only with CGNAT addresses for those users but we are planning to also deploy v6 end‑to‑end by the end of this year. There was a press statement about it.
JEN LINKOVA: Okay. Questions.
AUDIENCE SPEAKER: Thomas she ever. Just one question about how many users already use now IPv6 in your network, just percentage, or is it just a plan? I didn't ‑‑
RINSE KLOEK: We are still at the beginning of the implementation. So we are in a test phase, in a test phase we have about 50 users right now. In the new part of the network we have some old network, but it's often other part of the company but currently we have 50 users as mostly are test users. So, they are friendly users that just request can you enable it already. Awed add the other thing is okay, you mentioned the vendors but my question is you say bypass internal services we're doing CGNAT, but you don't mention that IPv6 is also a much much better bypass for it. And the Google box NetFlix and other way also in degrees a burden of CGNAT gateway, and yes, I would just comment.
RINSE KLOEK: True. Maybe I mentioned it not very good. But in the end v6 is the way to go. So if in an ideal world all content will be v6 and all users will have v6 ‑‑
AUDIENCE SPEAKER: But you do the work twice if you implement the by passing for private IPv4 space, and then additionally, later, IPv6 bypass, you save work in that case.
RINSE KLOEK: True too. But because of the v6 implementation it will take a lot of time. We started with ‑‑ and it was a very simple change so it was a change of just some communities and some routing, and so now we have to bypass via the direct connection to the caching box but if the next step we have fully deployed v6 then those bypass options are not used invalid more. So it's a temporary solution and for now it can save us the load on the CGNAT boxes. Add add I think your talk is like ten years ago, sorry.
RINSE KLOEK: But this is the reality.
AUDIENCE SPEAKER: The reality is in France, Belgium and Germany have 80% of IPv6 run via private users.
RINSE KLOEK: I agree. But the battle I'm doing now is within my company, so I need to convince people doing it so it takes a lot of time. I am convinced but I am just the engineering implementing it and I get a project to do that. So, yeah, I have to fight my dealts. But I agree. True.
JEN LINKOVA: Better slightly late than never.
AUDIENCE SPEAKER: Brian. You are mentioning IPv6 deployment in the core so I'm just wondering did you deploy it in the core as well or do you do like a 6 VP since there is no basically business case to deploy IPv6 in the core compared to say, you know, pushing down a CGNAT?
RINSE KLOEK: The new core net was fully dual stack. I did not hold TTL implementation. Everything was dual stack from day one, also the access network. Yeah. Is that answering your question?
JEN LINKOVA: Just in case I am cutting the line after people on the line currently.
AUDIENCE SPEAKER: Petter heather. I wanted to comment on one of the other commenters (Hessler) mentioned and that yes the best time to deploy IPv6 was 20 years ago. The second best time is now. And that unfortunately by discouraging people for people being so‑called late you are going to discourage IPv6 deployment in general rand it is unfortunate but it is a reality that many businesses will not prioritise IPv6 for a variety of reasons. But when they can, when they can encourage it we, as a community, should definitely encourage this adoption. Thank you.
RINSE KLOEK: Agreed. One last comment it was also the goal of my project like if you do CGNAT, you will have a business case for v6. It's not always a straight line. So some things you need to take some other paths but at the end I think now the company sees the urgency, so that was, that was my message.
JEN LINKOVA: Okay. Two last, two more questions online.
One question from Brian story: "How long has it taken for you to reach this stage? And at what point do you envisage IPv6 will be document for your customer base?"
RINSE KLOEK: Like I said my company has made a press statement I think this month that they will plan to do IPv6 implementation start the migration by the end of this year. So they said very decent goal so they published a press statement so they make it also very important. So, yeah, I think at the end of ‑‑ every new user will get dual stack and also the migration will start. I am hoping that by the end of this year we will have, for example, 20% and next year I think we will go to maybe 80 or a hundred percent, that's my goal and also the company set that goal. So I think it's now very positive. So...
JEN LINKOVA: Okay, we have the very, very very last question from Peter. "Can you elaborate a bit more on why you filter the IPv6 releases? Did of this any consideration to rapid commit."
RINSE KLOEK: About the release message.
JEN LINKOVA: Can you elaborate a bit more on why you filter the HTTPS releases in did of this any consideration to rapid commit."
RINSE KLOEK: I don't know exactly the question but with rapid commit I think you are release. We block the release message like the DHCP 6 server we relack on the release message it will always rotate the prefix and that's something we don't want. But I don't exactly ‑‑ understand ‑‑
JEN LINKOVA: I can guess you can follow this offline. So thank you very much for presenting your story. And the next presentation is William.
WILHELM BOEDDINGHAUS: Hello everyone. Good morning. So, I see the mic is working.
We always have the question why do enterprises do not implement IPv6? So why they are so reluctant to work on IPv6 and I think there are many reasons for this. We see a lot of IPv6 in the service provider field, as we now lender in the last presentation for consumers, but the enterprises, well, mostly they do nothing. Some of them do, but many of them do not.
So the question S. Why? And I I think we need to give them clear guidance on how they should implement IPv6. We have too many options, but they have no idea where to start and how to start.
So, what do we see today in the IT departments?
We have a lack of IT personnel. In Germany alone we have 130,000 positions in IT. The whole bandwidth from ASP to networking and I think we have roughly 500,000 open positions in the European Union and another 500,000 open positions in the US.
So, we have no one who does IT or at least we have not enough people who do IT. And in Germany the number of IT students is even decreasing. So I think IT is more important but we don't have young people who start in our industry.
Many enterprises don't even from a network department any more. Sometimes they lose the last networking guy and then they are without anybody who can start an IPv6 project because usually IPv6 is started in the network, we all know NIX is an IT project not a you recollect inning project. But in many cases it the started in the ‑‑ by the networking guys, and they have to do the first steps.
And the IT departments have no time. They are drowning, really drowning in their daily work so they have no time for new projects. Nobody of these IT guys go to their bosses and say hey, we have free time. We want another project. Please give us something we can do. They have enough to do.
And I have given a lot of training on IPv6, so some companies try to start. But have you ever done training in an enterprise when you go to the enterprise and don't get the guys out of their building? They are working on the daily stuff during the training. They don't even link to you as a trainer. How much did you learn in the evening? Nearly nothing and this can not start a project of course. And they have no time for reading. RFCs and RIPE papers and most of the time they don't understand them because they have so many tasks and so many projects to work on they are not just the networking guys who start with the network, they have ‑‑ they come from all departments and they have to do all of the IT.
So, the lack of knowledge. This is of course one of the biggest problems in the IT.
And if you go to a large enterprise and they start a new project. What are they doing? They inform every department. A department that is not informed of the new IPv6 project will not work with you. There are no technical reasons. This is just human interaction.
Okay, it is like this. Then every department gets a time to think of their requirements. So, everyone should bring in their requirements for the new project.
And the third step is that you collect the requirements. And we went to the security department of this company, and we asked for their requirements about IPv6, and this is what we got back. An empty sheet of paper. It was not that they had no requirements, but they hadn't thought about any of the requirements.
So, well the department was not unwilling to do things. But they just had no idea because they have never thought about IPv6. They had no prior knowledge of IPv6. They had no time to learn about IPv6. So this is why these projects don't start. What they already knew is that they had to put their IPv4 policies to the new IPv6 project. The first thing they told me yes we have to scan the networks from the first address to the last. I told them this isn't feasible with IPv6 because you need a few money thousands years per 64. But we have to. Our policy tells us to do.
So, this is where enterprises start with their IPv6 project and they have no idea, no real idea how to start this, because we have too many options that they could follow.
When they try to start, they say we need an address plan. And the tests will be dataseted from the beginning. And we better make no mistakes because it can't be changed in the future. Of course it can be changed but they don't want to change it.
They distribute all of the IP addresses they have evenly about, in their departments or subsidiaries or maybe even countries and they want to use the unique local addresses because they are like the IPv4 addresses they are used to.
I had one customer who wanted to distribute all of the addresses evenly to all of the countries of the world because they had sudden RIS in many many countries and said okay, put them on the map and (subsidiaries) and every country gets IP addresses and they had the same number of IP addresses for Lichtenstein and China. Do you think that is a good idea? And what is about new countries? Well there will be no new countries. This was the summer when there was a referendum in Scotland to become independent or not. I said well it could happen in a few weeks time that we have a new country. So what are you doing with your well distributed list?
.
No idea. So this doesn't work out, but this is the way the enterprises, many of the enterprises start to, start with an IPv6 project.
So, no idea how to start. Empty sheath of paper.
So what we, as a community, what can we do about this? We right papers. RFCs, best common operational practices. We have got the tendering document RIPE 772, and all the readers can learn from these papers. But do they? I suppose not.
If you look into these papers, this is from the RFC 9099, the security operations RFC from chapter 2.2, and there is four lines of text and we have three references to other RFCs. This is a perfectly well done document. And I think it is very useful, for me as a specialist, but not for an IT department who is trying to start securely with IPv6. They will never read it. And they will never understand it. Because they don't have the knowledge.
So these papers are usable for them in the way of the project, during the project at a later stage but not for the beginning. But we have these documents and this document is excellent, no question about that.
So, the RFCs define standards but they don't read them. Some of them are informational but they don't understand them. Some have best practices, but they are not put into operations because nobody reads them, knows about them so the RFCs are quite useless for starting an IPv6 project in an enterprise with guys who have no time to start the project.
From my experience, this does not work. Another example is you tell them hey you can use the M flag in combination with the A flag and you get what you want. They have no idea what they want but you can get it anyway.
This does not work out.
So, from my experience what can we do? We can give absolutely clear guidance how to configure the network. And get rid of many, many options that they have but they should not use.
I will give you a few examples and then we have sometime to discuss and collect ideas.
If you connect two routers, give them the rule user 64, set it aside, take a 127 out of this, give the router the addresses A and B because we don't want to use 0 and 1, 10 is so unreadable, then you have A and B and this is what you do with your routers. This is the path an admin can follow. Tech okay, I do it like this, now it pings perfectly well.
Then they know what to do. Clearly guidance.
For loop back addresses, okay, take aside a /64, take all loop backs out of this large address space and use the 128 per device. This is something an enterprise admin can do. If we give him or her 15 different options say yeah you might consider doing it this way. They have no idea how to start because they have not the knowledge to do this.
BGP:
Many of these enterprises have never used BGP before and today we are telling them get an AS number and get your own IPv6 prefixes because you don't want to use the provider addresses any more, renumbering is nothing, you want to do ever again.
They have never done BGP before. So they are struggling. Then they go to today, modern world, and get an example and then they use documentation prefix in the production network? Have you ever seen this? I think so. They have no idea how to filter. Do they even need filtering? . They have many many unanswered questions. But we can give them a list, and say okay, this is the prefix list you can use. Copy and paste and this is at least the start for your security. Then you have to filter your own networks as well that you don't announce your network to the outside world. But this is clear guidance and then the operators know what to do and have a start.
Of course when they gain experience, they will odd even modify this but for a start if they have no idea what to do, this could be a good idea.
The same is true for SLAAC and DHCP v6, they are not idea how this works together. Today most enterprises go for dual stack, this is just a transition technology, but they do not. It doesn't take much time, it's easier, it doesn't take many resources, they don't have to change the network design, that is quite important for them because they have worked with this network design for the last 20 years, so why change?
And they want to use DHCP as a central authority for IP addresses, they don't want to use SLAAC because they are used to it, they know how nature name servers work, and their DHCP servers and name servers work, they don't want to have any changes.
The clear guidance helps with starting the project quite quickly, saving costs, saving time an this is important for the IT departments, because many of the companies still see IT just as cost. They rely on it in every aspect of their business. But it is not an asset, it is cost. We should help them get security right by giving them pre‑configured filter list so that they have a starting, a good starting point and even if they use this filter list, then they are not optimal for their network, they still have more security than just an open our interface, and then we help them with getting IPv6 up and running and then they can get some experience with this, and then we get IPv6 implemented.
So, the question that I have: Can we, as a community, provide this clear guidance to the enterprises? We have best common operational practices papers, but can we agree to something that is suitable for enterprises? It's not optimal and there will always be corner cases and we are engineers and we know about the corner cases and yes, sir there could be sometimes a little bit problematic but if they have nothing, they don't even start the IPv6 project, and we wait for them to start it. So what can we do? Can we agree to this or is it just a stupid idea what I presented here and we can do nothing and we still give them RFCs they don't read?
.
Thank you very much.
(Applause)
AUDIENCE SPEAKER: Peter Hessler. I really like this concept. I have deployed IPv6 in a number of networks. I have been unable to deploy v6 in a number of networks for a variety of reasons. And one thing that I ran into is that it's, it is difficult if you're unfamiliar to actually deploy this. Like, give me, give a starting point of something that a lot of admins can copy and PACE. That definitely simple nice a lot of this. And you're correct, there are many ways to do this. But there should be an opinionated document that says this is one way how you can implement it. I think that would be hugely beneficial to a lot of enterprises.
AUDIENCE SPEAKER: Marco difficult it's, SIDN, thank you for this presentation. One thing that strikes me is when we get new colleagues in our company, they still don't know about IPv6. One would assume that it is in the curriculum of universities and schools. I'm not sure how that situation is in your country, but would you agree with me that we should also work on getting IPv6 knowledge into schools, into universities so that young students have at least some knowledge?
WILHELM BOEDDINGHAUS: Yes, do I agree and if the situation is bad in your country, it is even worse in Germany. They are still learning IPv4, the Professors in the universities, not all of them, many they have the script and they have used it for the last 20 years and they will use it for another 10 years, they will retire and then maybe something changes. So, we have to educate, yes of course, we have to do training, we have to educate young people and schools and universities should have their part in it. But they haven't.
AUDIENCE SPEAKER: I have a remark regarding IPv6 enterprise because I am the network guy who is pushing the IPv6 in my company and it's not only enterprises who have to be feed for this, but also the ISPs. On one occasion we had an uplink from the quite big and famous ISP, they provided us with /48 sub‑net and expected that all hosts in this /48 IPv6 sub‑net talk to their gateway without sub‑netting. And it took me two weeks to explain to them that it's not the way how it should work. And to establish a point to point sub‑net over the one link and to route this divided subnets to them. So it's unbelievable how even big providers don't understand how it works.
WILHELM BOEDDINGHAUS: Yeah, that's absolutely true. I think most of the providers, or many of the providers have a good IPv6 network and have a good understanding of how IPv6 works and the reason is the network is their product. But the enterprises build cars, build ships, sell insurances, so they don't have IT as their core business or networking as their core business. And they are struggling even more. And please, usually in many countries, don't look into the public administration. They are struggling even more. Not because they don't want to do it, they just have no resources at all to get IPv6 up and running.
AUDIENCE SPEAKER: I'd also like to pick up to the previous speaker here, I think his name was Marco, the university curriculum is about IPv6 are absolutely atrocious. Like to the point that they are misleading and maybe they are better if you didn't learn about them. Because as far as I have lender, I am an IPv66en /THAOUSist, they are big ‑‑ I am philosophical differences between how IPv6 should be done and how IPv4 and that is not taught anywhere. If you are talking about guidance, maybe universities should get in first.
I'd like to pick up on so IPv6 in enter rises, well money speaks in business, so, I think that although I'm an IPv6 enthusiast, dual stack enthusiast sounds like cost and added security vector, training personnel sounds like cost and like in a time when enterprises are moving as much as possible, I think we're not getting further even if you are giving them the configuration or guidance, because many of them are not capable. For those that do it, for those Internet providers that go IPv6, I think they can get a stellar best in class IT engineers. But maybe what I would focus on if we want IPv6 in enterprises to be, to be picked up, is on the benefit, on the incentives and I think in IPv6, a lot of incentives are not there at the beginning, there is future approving which should be put there and I think for example in case of mergers, a lot of resources can be of mergers of enterprises a lot of money can be saved if you can guarantee hey at some point when you are merged with somebody, you'll have networks that are compatible, that you won't have to re‑number or do other things so. I think that we should put a lot of focus on those incentives, find them, make a big list and try to sell it to the enterprises, to their monitors.
WILHELM BOEDDINGHAUS: You can try to sell it to the monitors but they say as soon as we have a major ‑‑ as soon as we get the new company in our group, we will do it, if we have a merger. They will not do it in advance. This is what I think about it. What universities, yes, they have to do more, maybe we can try to educate them, but they are the education institutions aren't they. They should pick up the modern protocol that has been designed before the year 2000.
.
And IPv6 as a project like the year 2000 project or the project that you have the euro as a new currency, you don't gain anything from, you just have to do it sometime. One was in the calendar, the year 2000, the other in the law, the euro. We are missing that but we can't have that of course.
JEN LINKOVA: Okay. I am closing the line after people who are already included, so that's fine. Gert.
GERT DÖRING: IPv6 sort offen /THAOUSist. I agree with all previous speakers that there is hundreds and thousands of place where is we should do better IPv6 but I thank you for bringing the enterprise issue to us. Because I think we are all network engineers and have, well many of us have little contact with the enterprise world. So I totally agree with your problem statement that there is too many solutions.
So, if you pick ‑‑ if you look at the technology, there is ten solutions for every IPv6 problem and picking the right within for your enterprise, that will not come back and byte you in two years is a hard challenge. (Byte)
.
I think on that question we should try, and I'm willing to help with that. I am not going to drive the effort but I'm which will to provide input.
It will not be easy because when we did the BCOP an how ISPs should do things, the amount of fights we had in that group on what the correct solution is picking from the IETF options, it wasn't hard ‑‑ it was hard to agree on this is the correct thing. There is a reason why there is ten different options, because smart people have come up with things.
But we should try.
WILHELM BOEDDINGHAUS: Thank you very much, because I think it is needed and we can suggest something to the enterprises. Maybe we should, if we do as a community decide we want to do something about this, we should try to find enterprise admins in our group. So maybe we are tacking about their problems without having them in our group. This is maybe not a ‑‑ not one of the best ideas.
AUDIENCE SPEAKER: Benedikt Stockebrand, talk for myself without the Working Group Chair, and I'm also working together with William on some things. Just for disclosure. There is one thing about academia and training people that we shouldn't forget. Most people don't get any network training at all in ACK demand a before they get into networking because (academia) they have got a Ph.D. in biology, archeology, whatever, but nothing to do with IT. Some of them pick up things the way they do, others don't. But it's not feasible to assume that the academic world will take care of these things completely. It's still going to be to some degree our job to spread the word at a level that people without a proper IT background can actually handle, because those are the people who run networks and IT in a lot of organisations, a lot of enterprises.
JEN LINKOVA: Actually, I see a comment online which I think is related to what Benedikt said.
"A remark: Ditching IPv6 in university, in the case of Spain, is limited to mark in demand competences. When enterprises improve in IPv6 adoption it will be few demands that university will offer. Nowadays IPv4 is basic as there is not much time available anyway."
.
Gert are you still queue? I would love to have this discussion but we are running out of time. I am sorry, so we still have two people in the microphone and one online. Let's be quick.
AUDIENCE SPEAKER: Peter Hessler. When I have tried to implement IPv6 in some locations I ran into the problem of the ISP that we were using refused to have IPv6 available to us. In some cases they didn't have any IPv6 on their entire network at all. In other cases, the most funny one I found was a residential user of this ISP got IPv6 automatically. The business users, they refused to provide the v6 to them. So, your connectivity, even if you do implement IPv6 internally, well I guess you're setting up a tunnel to Hurricane Electric and who benefits from that?
WILHELM BOEDDINGHAUS: If you are in a city, then you can maybe choose from a lot of providers. But if you have your company in rural area, then it might be not the choice of providers.
AUDIENCE SPEAKER: I will give a spoiler. This was in Berlin.
WILHELM BOEDDINGHAUS: I have a good idea because I live in Berlin as well, I have a good idea who this provider is. Well we are not going to bash!
AUDIENCE SPEAKER: Hello. I am going to I have been opinion an IPv6 pusher for 25 yearsnd an I know how hard it is in Denmark where it's being ignored. I am going to take on my teachers hat because I am in the Copenhagen school of design and technology. Thank you very much because I think exactly what you are proposing is what we need and I agree with all the variants but it's very, very fine to leave out a lot of datasets that people can discover for themselves later but we need to get people running and from my own point I also have a call later on for resources that can be helped for bringing in the newcomers to the field of networking, DHCP is not being taught anywhere as much as it should be and its with old resources and without IPv6 and we are the IPv6 Working Group, so please if you could send me hlk [at] kamsa [dot] org, send me where you are best resources for newcomers and beginners in IPv6 and TCP IP and I will definitely be open with the writing or reviewing and working on those documents like you have proposed.
WILHELM BOEDDINGHAUS: That you know very much. If you have any more questions, I will be around ‑‑
JEN LINKOVA: One more last remark online from Marcus. "A remark from my side. I am teaching IPv6 for about ten years now with my university. Questions I get all the time from my students why we have to learn about IPv6 when nobody is using it or it is spreading too slowly?"
I don't know, I have almost no IPv4. So, I am surprised. But, I guess it's just a different part of the world, right. There are places where there is no v6, there is places where there is no v4, so...
WILHELM BOEDDINGHAUS: If even the young people today say we don't want to learn it, then we are lost. Thanks for your attention I will be ‑‑
JEN LINKOVA: Very last closing remark. I am abusing my power as a Chair. I see good discussions so what's the next step? What's the action item?
WILHELM BOEDDINGHAUS: I think we should ‑‑ I'm going to put a mail on the mailing list and then we will see how, if we can find a group who wants to start with this. Maybe we can avoid the fighting of the BComm for service providers. But let's start, let's try, whether we get something together. So I will put something on the mailing list of this Working Group.
JEN LINKOVA: Thank you very much for doing this, highly appreciated.
(Applause)
.
Okay, the next presentation is from fern and owe who is online.
FERNANDO GONT: Good morning everyone, I will be doing this presentation on the implementations of IPv6 addressing on security operations.
So let's start with the motivation for this presentation.
We know that there are a lot of differences between IPv6 and IPv4, including when it comes to addressing, but normally these differences or these differences are not very obvious outside of IPv6 circles. And I'm talking it could be like security groups, it could be Cloud operation groups and others. Now what normally happens in these cases is that if people or these groups they don't know the differences, what normally happens is that they end up applying IPv4 practices and that normally leads to supply optimal scenarios. This presentation has been motivated by interactions with some of such groups.
So, let's provide some background because these presentations essentially the title even mentions like security, operations and IPv6 addresses so let's get everyone in on some of the concepts before we jump to the actual content of the talk.
First of all, what do we mean by security operations in well there is a lot of things that security group does on a regular basis. But, you know, for the sake of the discussion, I will try to take just two of them which I believe they serve as good examples of how IPv6 addressing will affect these operations.
So I have selected these two. One is that enforcing address space access control lists and the other one is about doing network activity correlation. Obviously we all know and we all work with ACK owes, we are going to divide them into two classes, al owe list and I block list. One we want to hello access to some information resource from some specific prefix or address. Where as for block list it's obviously the opposite. We want to block (allow) from a specific prefix or person.
When it comes to network activity correlation, this is something that for example you might need to do doing incident response. So you are going through a lot of logs and you try to figure out whether different kinds of activities have to do with a the same actor. So these are the two main groups of activity or parts of security operations that we will be let's say talking about during this presentation.
Then, you know, in one of the first few slides I was talking about differences between IPv6 and IPv4 and obviously because of the title of this presentation, we are going to just talk on differences when it comes to addressing. And there is a number of differences in here. First of all, we have to look at the properties of the addresses. In IPv6 we have multiple properties and multiple combinations of such properties. For example we have different other scopes like globalling local and so on. Stability properties, and the next one is kind of related with the previous one like intended usage for the addresses like temporary addresses that are meant to be uses for outgoing communications such as other addresses such as stable addresses are for incoming communications.
Key difference between IPv4 and IPv6 in address it is you have an IPv6 host using multiple addresses simultaneous, which normally doesn't have IPv4. And another one which is like makes a big difference in the topic that we are going to discuss today is that in the IPv6 case, users normally have control over a big part of the address space, or over a big IPv6 address block, at least a /64.
Now, when doing security operations, one thing that is like key and super relevant is, you know, what's behind an IPv6 address or an IPv6 prefix? Like, when I look at an IPv6 prefix or an IPv6 address, what is that prefix or address identifying? Well, if you think about it, like, you know, obviously multiple addresses may map to the same system, so the same cost. This is, you know, this is probably expected because we were discussing before that hosts typically configure multiple addresses by normally from the same /64. But in a lot of cases, you know, a user or they could also control a larger address block for example such as a /48.
Now, one key takeaway from here, you know, that relates to security operations is that different IPv6 addresses does not imply differentest hosts or different actors. Then, on the other hand, if we think about a single address, that doesn't necessarily mean or doesn't necessarily identify a single host. Why? Well because whether we like it or not, you know, NAT PT or different versions of NAT are not uncommon. If you look at a typical Kubernetes application, one the things they do, vendor not to be named here S doing IPv6 ULAs with NAT PT, so NAT PT is not unusual. So the take away from this is that if you are doing for example network activity correlation and you are looking at the single address, well the single address does not necessarily mean that it's the same cost of the same actor. All of these aspects of course are key when doing IPv6 security operations.
So, what we'll try to do now is to try to go through each of the topics that we were discussing before, allows and block lists and network activity correlation and try to figure out first of all what are the some of the challenges in there.
And then try to, you know, come up with something that we could do to, let's say, deal or overcome those challenges.
So, the first one is with ACLs corresponding to allow lists. So, in a lot of cases, would say that's probably the case for most general purpose host operating systems they ship with temporary addresses enabled by default. So that means that essentially addresses change on a regular basis. They could change for example once a day. And normally the addresses will be selected from the same /64, okay.
So, the question in this regard is if we have a network and we have a device in there and for some reason we want to, you know, configure an ACL and include that system in Anna allow list what should we do? Because if we were to just simply specify a/18 which is analogous to let's say specifying a /32 in IPv4, well if the host addresses are changing all the time, well maybe we are not going to get what we would expect. That's the challenge with configuring the allow list.
Now what happened with block lists? Well if you are doing security operations, you normally have like a number of figuring of different sorts. That would be your SI EM platform. People use fail 2 ban or use IP reputation service to say somehow learn about addresses that are supposed to be malicious or addresses that are supposed to be offensive at the time. And quite often, well security groups do is react to those fields of information and just block the offending address for some period of time. The question ‑‑ in IPv4 obviously you do that for a /32, with whether etc. Perfect or not that's a different question but that's what we normally do for IPv4. Now the question is in the IPv6 world, what should we do? Because the analogous to 2004 would be to just simply block a/128.
With you, the problem there is that, you know, based ‑‑ if you think about, you know, the fact that an something will normally have control over a large portion of the address space, like at least a /64, it's tricky for a number of reasons. First of all the actor could actually change the addresses over time so that you actually let's say add a lot of ACLs, a lot of block entries in your ACL, so that for example you exhaust the number of entries that you can have in that ACL list. That's one option.
The other thing is that an person could do is change the addresses all the time so that by the time that you get to consider or you get to infer that an address is being offensive, it's doing malicious activity, the user has already moved over to a different address so the address you are actually at is not in use any more. This is super doable. If you are on a sub‑net that is using a /64, it's super easy to do that, so that you can circumvent like the typical security controls that people apply based on the security, the IPv4 security practices.
So, it's a challenge what you block.
Most likely, if you just try to block a/128, things are not going to work as expected because a skilled actor will just skip and hop through different addresses an the add query will be in a position to circumvent your block lists.
Then there is the topic of network activity correlation. And you know based on the background that we were discussing before, it's clear that it's a non‑trivial exercise because you could have like multiple systems behind the same/128. This could be like multiple ports in a cluster for example, or like multiple users before some sort of NAT. Again, just being the messenger here, I'm not saying that this is not or not, it's something that if you are in a security team you have to deal with these facts. But you also need to deal with the situation in which the add Kerr simply has control over at least a/54 but it could also happen the add Kerr could also have control over a /48, and this is trivial and the add Kerr could also have control over multiple lash 48s or anything in between could happen.
So, what we are going to try now is to to try to discuss like possible advice, like possible things that we could do to actually, you know, circumvent ordeal with these issues.
So, first topic is we're going to try to go through the same ones that they were before like allow lists block lists and network activity correlation. For a low list there is not a lot to do so the options are option 1, use stable addresses, and if the question is okay how do you get to use a stable addresses? .then the options probably are do manual configuration, do the DHCP version 6 which somehow gets you a stable address or use SLAAC and disable the use of temporary addresses. If you are in an enterprise are group policies. Then you if you do this you make sure you are using stable addresses then you could resource certificate to specifying allow list as/128s.
The other option is to embrace the use of temporary addresses and in that case, somehow you have to segregate systems into different subnets and then your ACLs would be /64, with recent development in the idea it is to be expected that for example hosts or mobile phones will get each a /64. So if what you are planning to do is to block a single system or be successful at blocking that system, even at the expense of side effects, then you should be prepared to block a /64.
What about block lists? Well, there is a number of challenges in there. First of all, I was describing before since the add Kerr will typically have control over a large portion of the address space, you should expect that the add Kerr might hop through different addresses. So you should be prepared to configure ACLs with different granularity, not just a/128 but even in other granularities. And if these rules, if these ACLs are being dynamically generated you probably have to think about dynamically aggregating these ACLs and also changing the lifetime of the ACLs that you can configure based on the grant large.
This next slide tries to illustrate what we mean by this. This is by no means ‑‑ so the values that you see here are by no means values that we are recommending. You can let's say we have different aggregation levels, 1, 2, 3, 4 and so on and the idea is if you start configuring/128 ACLs and if eventually you can aggregate ten of these lash 128 into a single /64 you do it. As you aggregate the ACLs into a more porous granularity you want to reduce the lifetime of the ACL because the side effects are going to be larger, so when you block a single/128, you the lifetime of the ACL for one hour, but if you now specify a /64, you might want to use, I don't know, five minutes, 30 minutes, whatever that is, and if you further aggregate that, then you would probably want to review the lifetime. Obviously this is tricky because as in the same way as the ACLs, its granularity gets course err, you can effectively like block the add Kerr if the add Kerr is hopping from one address to the another then the possibility of a side effect obviously increases.
.
So that's for ACLs and block lists. Now when it comes to network activity correlation, at the extremely varied list, the expectation here is whatever tools you are using for network activity correlation, they should be able to somehow mark activity as being related not just on a per address space but also on a per prefix space. It's not just if you find the same IPv6 address, the same/128 in different parts of the logs, it's supposed to be the same actor but you should also be able to specify well I want to correlate network activity based on a /64 or a/whatever.
So, this is a simple overview of the issues. Consider this presentation brainstorming. I believe that a lot of these things there are like open topics. And if we want to jump into, you know, conclusions before we jump into the Q&A part of the talk, I would say that the differences between IPv6 and IPv4 addressing features, they have concrete implications on security operations. In a lot of cases these differences are non‑obvious outside of IPv6 circles. That's normally the case with security operations groups, they normally have no background on IPv6 so the expectations are that well there's this thing called IPv6, we guess that is it's like IPv4 with longer addresses so we'll just do the same. This also happens in Cloud operations group. Probably everyone outside of specific networking circles. And the obvious outcome of this is that if these groups with not aware about the datasets they will end up applying IPv4 practices, and, you know, the outcome of that is going to be like at least suboptimal it not like a complete failure.
So, the take away I guess is that security operations require changes to embrace IPv6 or require at least to give these differences in IPv4 and IPv6 addresses like some consideration.
So that's it for the presentation. So, do we have like time for questions, comments?
JEN LINKOVA: About ten minutes for questions. So...
AUDIENCE SPEAKER: Fernando. Very good presentation. V6 security is a bit messy. One thing to mention, very recognisable. One thing I did also a lot of CPE security testing and then you have to only the local LAN you are testing and one thing I encountered is that sometimes services are listening on local addresses and if you copy your v4 test to v6 test you might miss that. I have seen that happening in the field. That was one of my additions. Good presentation.
FERNANDO GONT: That happens a lot like for testers, like I'd say like in the vast majority if not all cases that I have dealt with, obviously they don't know what ULAs are, it's just IPv6 addresses to them and I guess they expect them to be globally reachable which is not the case. So, yeah, there is a lot to go in there.
AUDIENCE SPEAKER: Benedikt Stockebrand. There is one thing to keep in mind. If you use IPv6, you have another option when it comes to a lot of security things called micro segmentation, that really solves a bunch of problems that you were addressing at the root, and the only downside, if you want to call it is that, is you have to let go of your IPv4 way of thinking and doing things. But if you go that way and don't use dual stack where it really forces you to consider the constraints of both v4 and v6, you can make your life much easier than trying to do things together. I it's just an option to keep in mind. It doesn't always work so what you said was really valuable, but if you can take the easy route, it's the easy route.
AUDIENCE SPEAKER: Peter Hessler: One thing I have noticed when translating a v4 firewall ACL rule set over to IPv6 is very common to block ICMP, so of course if you are not aware of it it's very common to block ICMP v6 and then about ten or 20 minutes long you are you realise you have no longer IPv6 because you have blocked all neighbour discovery. So there are some really critically important things that you have to be aware of when deploying this from a security perspective and make sure you don't over zealously do the same thing that happened to work because address solution in v4 was not on the IP layer. Whereas in address solution in IPv6 is on the IP layer.
JEN LINKOVA: Any more questions, because I might jump with a couple of could you please go to slide 10 I think. So, I just wanted to ask, so actually I think why, if it's not your network, I mean if you are blocking someone else why would you care? You should care about blocking something which could be attributed to a user if you assume that IP allocates/26 per home network, I think it's safe enough to assume we block the owner of the address and it doesn't matter how many addresses they have, right. So, I think it would depend on the address allocation policy of the other side. Because it should be their ISP dealing with like how granular and what to do with that ACL, right?
FERNANDO GONT: Yeah, I say most of this presentation came up with this. We were in a meeting with some fairly large content providers and they were discussing how they were doing some of the security operations and they got to talk about all the ACLs and I asked the questions the granularity for the ACLs that you use, assuming they would use something like /64 they were like what do you mean? I said what's the granularity that you use in they said they just block the others. So, the thing is that I agree with you, the problem probably is that the groups that are like trying to implement these controls, they come from an IPv4 mindset, and all of these datasets that might seem like obvious to this group are not obvious to the people actually doing like the security operations.
JEN LINKOVA: And I think that's actually much worse right because of CGN for example you might block and address and you block a lot of innocent people. While with the v6 you have much better chances to actually block like Openflow actually malicious users.
So, I have just two last comments. I think it would be nice to be clear about what the actual difference between v4 and v6 because I think a lot of those problems actually apply to v4 as well.
Any more questions?
.
Okay. Great. Thank you very much. Okay, so we have one ‑‑ like ‑‑
.
(Applause)
.
I think Benedikt wanted to make some announcements and we have some very last moment another business after Benedikt announcement I guess. So Benedikt.
BENEDIKT STOCKEBRAND: Just a very quick one. My term as Working Group Chair ends with the next RIPE meeting, and I won't stand for another term. So, just so you know and you have sometime to think about if you might be interested in taking over. I thought it was a better idea to tell you here in person than sending things over the mailing list like four weeks before the next meeting and things get a bit hectic. It's not too much work or too annoying or whatever. It's just it doesn't fit well with what I am currently working on so I think it's time to pass on the hat to whoever is interested. Just so you know and have sometime to think about it. Okay. That's it.
JEN LINKOVA: We are still going to see you on stage next time. Please think about it. Talk to monitors if needed, volunteer.
And we have the last minute other business. I don't know what it is, it's a surprise. Speaker speaker hello, this is Ondrej from RIPE NCC from the tech team. And it's sort of my responsibility the main meeting network is running sort of IPv6‑only, it's IPv6‑only for 78 percent of participants here, we discovered some interesting bugs, actually I talked about this in the last meeting so we can look into the archives. I just want to tell because here are the IPv6 people. Please go and test and find out what's wrong and report it to the vendors of the machines that are going wrong, because it seems that yeah in the lab everything works and then when 700 people connect, it just ‑‑ you see very strange things. So that was the only thing and more on Friday in the technical report. Thank you very much.
JEN LINKOVA: Thank you very much.
Actually, very nice introduction because I am going to talk about something like that next time.
So thank you very much. Before you leave the room, please rate the talks. Nobody leaves the room until you rate the talks. We need to know what you like and what you don't. Otherwise we are going to see the same presentations, or maybe something different, I don't know. Thank you everyone. See you in November.
LIVE CAPTIONING BY
MARY McKEON, RMR, CRR, CBC
DUBLIN, IRELAND.
