Recently Windows Azure had a nearly global problem with its Azure Storage system which resulted in real downtime for many services, see here.  This affected one of my personal projects that was in development, basically even after the fix was made I was not able to access any of my endpoints.

To correct this, I performed a fresh publish ensuring the option to remove files at the destination was checked.  This succeeded in restoring the service and I thought everything was fine.  However, over the next couple of weeks I noticed some issues with updating the service, however I lacked the time to fully dive into the problem.

I finally got time around Thanksgiving as I was grounded for the week from traveling.  The main problem was that any new changes to the code would not show up on the server, that is I would get 404s for my new endpoints.  At first, I thought this was a routing issues since I was using custom routes for the new endpoints.  However, when I updated an accessible endpoint to return “Hello World” I got the original result, which confirmed to me that it was not a routing issue (all of this code worked locally) but instead a problem updating the code.

Even after continued efforts to re-publish with the removal of destination files, no change was seen.  So, I downloaded the Publish profile from the website, imported it and extracted the FTP server, username, and password and plugged that in the FileZilla.  I then attempted to remove everything from my wwwroot and repository directories.  I figured this would bring the service down and allow me to forcibly push the right DLLs; it did not.

Even with all files removed, I was able to hit the endpoints I knew still worked.  Even when I published to those empty directories, I still got the old result.  Thus, the only path I had left for me was to delete my service and recreate it with the same name.

I had to be very careful doing this because I did not want A) to delete my database and B) I wanted the new service to utilize the same schema name in the SQL Azure Database.  Thankfully, this approach worked and the service is back online and I can confirm new changes are working properly.  This also had the side effect of fixing the problem with my connection string that I noticed a few months ago.

Speaking with MVPs and Azure team members, no one seems able to explain what exactly happened or why those DLLs would not update.  My guess is that the storage mechanism behind the scenes had to change in response to the outage so my old service was likely located in a bad area, but with recreation is now located in a good area.

Hopefully, I can get some clarity on what happened.  In the meantime, I hope this helps anyone who is having issues the same as me.  Though, I realize deleting the service and recreating it is not a fun option