DNS Stuff
      Back to home
      1. Internal RG Wiki
      2. Internal Documents
      3. DNS Stuff

      Can't Sync DNS in MCP for a new site?

      There are a few different reasons this can happen. Some of those are:

      • Weird DNS records were added in route53 when the domain was created.
        • Solution: delete all the records for the hosted zone in route53, except for the SOA and NS, then run sync dns in MCP again
      • You got unlucky and tried to sync dns while route53 was rate limiting us.
      • You got really unlucky and are experiencing one of the weirdest issues support has to deal with: The AWS DNS Dance. (This is the one we'll be talking about)
      So what causes us to have to do the AWS DNS Dance?

      If a client's previous platform or site host used AWS route53 to manage their DNS, and they also happened to be on the same delegation set as our nameservers are (basically if we are sharing the same "cloud" with them) Then what happens is that we are unable to create a domain in route53 because a hosted zone resource already exists on the delegation set (It's just in an account we dont have access to)

      How do we fix this?

      We create the domain on our alternate nameservers that are on a different delegation set.

      How the heckin heck do I do that? How the heckin' heck can I test this as a lone, unlucky support member? I don't have the powah.

      Well lucky for you, your AWS account should have access to the route53 api via the AWS CLI! The steps below will show you how to confirm if the issue is the AWS DNS Dance and how to fix this problem.

      If you arent comfortable using the command line then ask for help before trying to do it on your own.

      The first step before doing anything is: Double check everything to make sure we actually need to do this.

      Im not going to get in to the all the crazy specifics of what and why this can happen because honestly, DNS is boring. Instead Im going to show you what to look out for, and give you some examples:

      First things first: Check the nameservers of the domain using the dig command in your terminal.

       

      Wait a second, those nameserver domains look nothing like ours! They dont even have our two favorite character in them! This client isnt even on our nameservers bro, WTF?!

      Well actually, the nsX.rgnameserver.com domains are aliases that we use for ugly domains like these because they're easier to remember and give to clients.

      Oh! Okay. Those are probably right then. I guess its something that isnt the DNS.

       WRONG! ITS ALWAYS DNS!

      But also, instead of just assuming that those are right, since we're in our terminal it's really easy to check and compare the nameserver values side by side so there is no reason not to double check. It's almost like thats the entire reason we're doing this... wait...

      Anywho, to get the actual domains for all of our AWS nameservers run this command in your terminal:

       

      Oh wow! Thats weird... The nameservers that are in my DNS query are really similar to ours but not the exact same. But that 1004 nameserver domain is in both the dig output and our first delegation set. Why is the DNS query only showing one of our namerservers? Where did those other 3 come from?

      Last things first: I have no idea where those other 3 nameservers came from in our DNS query results. It really doesnt matter either TBH.

      What does matter is that both the results have that 1004 nameserver domain. This means that for some reason ns-1004 has records for the domain we're wanting to use, and those are being used for resolution before the records on ours are.  

      My current theory is that because our SOA record uses ns1.rgnameserver.com (which is an alias for ns-1896.awsdns-45.co.uk)  and "The other guys" are using the ns-1004 as their SOA, they get resolved before us because 1004 comes before 1896 (fight me if you disagree)

       This whole partial nameserver match combined with the fact that the site is down means the we gotta dance. A very special dance. The AWS DNS Dance.

      It's simpler than the texas 2 step. Only there are more steps (listed below for you to follow), its less fun, and there isnt actually any dancing.

      FIXING THE THING (along with another verification step)!

      1. Try to create the domain on our normal nameserver delegation set using the AWS cli. 
        • aws route53 create-hosted-zone --name=DOMAIN_WITHOUT_WWW.com --delegation-set=NFACWVCUTB90B --caller-reference=YOUR_NAME_AND_THE_DATE_NO_SPACES_ALLOWED
        • If you run the above command and dont get any sort of error message, then congratulations! You dont have to do the AWS DNS Dance! Go sync dns in MCP. Otherwise go to step 2
      2. 'Kay. You ran the command from step 1 and you got something like this: 
        Cannot create hosted zone with DelegationSetId DelegationSetId:NFACWVCUTB90B as the DNSName southfloridaishome.com. conflicts with existing ones sharing the delegation set
      3. You need to create the domain on our alternate delegation set. This is done with the same command as above, just with different parameters:
        • aws route53 create-hosted-zone --name=DOMAIN_WITHOUT_WWW.com --delegation-set-id=NT8H736FUOTAF --caller-reference=YOUR_NAME_AND_THE_DATE_NO_SPACES_ALLOWED
      4. Once you do this you should get a response of JSON that looks successful in your terminal. 
        {
            "Location": "https://route53.amazonaws.com/2013-04-01/hostedzone/Z0602860KC5SP1AY4KT2",
            "HostedZone": {
                "Id": "/hostedzone/Z0602860KC5SP1AY4KT2",
                "Name": "somerandomdomainname.com.",
                "CallerReference": "this_is_a_test",
                "Config": {
                    "PrivateZone": false
                },
                "ResourceRecordSetCount": 2
            },
            "ChangeInfo": {
                "Id": "/change/C02488792LS5O82UXT8U4",
                "Status": "PENDING",
                "SubmittedAt": "2020-03-24T16:20:20.825Z"
            },
            "DelegationSet": {
                "Id": "/delegationset/NT8H736FUOTAF",
                "CallerReference": "foofoo",
                "NameServers": [
                    "ns-1864.awsdns-41.co.uk",
                    "ns-966.awsdns-56.net",
                    "ns-339.awsdns-42.com",
                    "ns-1144.awsdns-15.org"
                ]
            }
        }
      5. Now comes the tough part. You will need to manually add all the required DNS records in route53 because sync_dns can no longer be used on this domain. Sync dns will break this domain. You can use this hosted zone as an example , Leave the NS and SOA records on the new domain alone everything else should be changed. 
      6. The final step is to tell the client that they will have to use different nameservers from the ones in our docs. You can get the nameservers to use from the JSON response you got when you created the hosted zone in your terminal. But just incase they are listed below as well:
        1. ns-1864.awsdns-41.co.uk
        2. ns-966.awsdns-56.net
        3. ns-339.awsdns-42.com
        4. ns-1144.awsdns-15.org
      7. Make sure to make a note that this domain is special in the zendesk ticket. Include the correct nameservers and soa record by copying them from route53. Make sure to explicitly state that running sync dns in mcp on this site will break things. So that any poor support person who is trying to figure out why the clients site is down after they synced dns will be able to see their mistake and have a quick answer for fixing it.