Sitemap.Xml

Concept 1: What is a Sitemap.XML?

A sitemap.xml is a file that tells search engines (like Google) what pages exist on a website and how important/fresh they are.

Think of it like a table of contents for a website — but written for robots, not humans.

Basic Structure of Each Entry:

[URL] [Last Modified Date] [Change Frequency] [Priority]

Example from Netflix:

https://netflixtechblog.com/tagged/kafka    2025-07-10    monthly    1.0

---

Concept 2: The Four Key Fields

Field 1: URL (Location)

https://netflixtechblog.com/tagged/machine-learning

• This is the exact web address of the page
• Search engines visit each URL listed here
• Netflix organizes theirs by topic tags (kafka, spark, redis, etc.)

---

Field 2: Last Modified Date (lastmod)

2025-07-10

• Tells search engines when the page was last changed
• Helps Google decide whether to re-crawl the page
• Format is always: YYYY-MM-DD

From the Netflix sitemap, notice:

/tagged/chukwa        → 2016-02-15   (old, rarely updated)
/tagged/kafka         → 2025-07-10   (recent, active topic)
/tagged/netflix       → 2026-06-29   (very current)

Key insight: Older dates = stale content. Newer dates = actively maintained.

---

Field 3: Change Frequency (changefreq)

This tells crawlers how often to come back.

From the sitemap:

/tagged/netflix          → daily    ← Most active
/tagged/machine-learning → daily    ← Very active
/tagged/cassandra        → weekly   ← Moderately active
/tagged/kafka            → monthly  ← Less frequent

Key insight: daily pages are the most important to crawl frequently.

---

Field 4: Priority (priority)

1.0

• A number between 0.0 and 1.0
• Tells search engines relative importance within your site
• Netflix uses 1.0 for everything — meaning all their tech blog pages are treated equally important

Note: Priority only compares pages within your own site, not against other websites.

---

Concept 3: How Sitemaps Reveal Site Structure

By reading the Netflix sitemap, you can learn a lot about the site:

Topics Netflix Writes About Most (by recency):

AI/ML:          /tagged/machine-learning  → daily updates
Recommendations:/tagged/recommendations  → weekly updates  
Databases:      /tagged/cassandra         → weekly updates
Infrastructure: /tagged/kubernetes        → weekly updates

Topics That Are Older/Less Active:

/tagged/gearvr          → 2015-09-24  (VR experiment, abandoned)
/tagged/oculus          → 2015-09-24  (old project)
/tagged/silverlight     → 2013-04-15  (obsolete technology)

Key insight: A sitemap is like a history book of what a company cared about over time.

---

Concept 4: Why Sitemaps Matter (SEO & Crawling)

The Problem Sitemaps Solve:

Without a sitemap, search engines must discover pages by following links. They might miss pages that are:

• Deep in the site
• Not linked from other pages
• New and not yet discovered

How Search Engines Use Sitemaps:

Step 1: Google reads sitemap.xml
Step 2: Finds list of all URLs
Step 3: Checks lastmod → "Has this changed since I last visited?"
Step 4: Checks changefreq → "How often should I come back?"
Step 5: Checks priority → "Which pages matter most?"
Step 6: Crawls and indexes accordingly

---

Concept 5: Sitemap Patterns You Can Spot

Pattern 1: Tag-Based Organization

Netflix organizes by topic tags:

/tagged/kafka
/tagged/machine-learning
/tagged/chaos-engineering

This tells us their CMS (Content Management System) auto-generates tag pages.

Pattern 2: Article URLs

Some entries are direct article links:

/java-21-virtual-threads-dude-wheres-my-lock-3052540e231d
/how-netflix-scales-its-api-with-graphql-federation-part-1

Pattern 3: Future Dates

Notice some dates are in the future:

/tagged/localization    → 2026-03-06
/tagged/software-engineering → 2026-05-29

This can happen when content is scheduled ahead or dates are set incorrectly — a common real-world sitemap issue.

---

Summary Table

---

Quick Mental Model

🗺️ Think of a sitemap like a city map given to a tourist (search engine):

• URLs = street addresses
• lastmod = "this building was renovated recently"
• changefreq = "this market opens daily vs. this museum opens monthly"
• priority = "definitely visit the main attractions first"