Explanation: Understanding DETEQTIVE DNS API

Understanding-oriented: Deep dive into concepts, design decisions, and the thinking behind the DETEQTIVE DNS API.

Authentication

All API requests require a Bearer token. Set it once in your shell and all examples will work:

export DETEQTIVE_TOKEN="your-api-token-here"

What is Passive DNS?

The Concept

Passive DNS is fundamentally different from traditional (active) DNS lookups:

Active DNS (Traditional):

Question: “What is the IP for google.com right now?”
Answer: “142.250.185.46” (current state)
Limitation: No historical context

Passive DNS:

Question: “What IPs has google.com used over time?”
Answer: Historical database of all observed DNS records
Advantage: Time-travel through DNS history

Why Passive DNS Matters

1. Security Research

Threat Intelligence:

Track malware command-and-control infrastructure
Identify phishing campaigns
Detect compromised domains
Map botnet networks

Example: A domain suddenly points to an IP known for malicious activity. Passive DNS shows when this change occurred and what the domain pointed to before.

2. Infrastructure Monitoring

Change Detection:

Monitor your own infrastructure for unexpected changes
Track competitor movements
Understand CDN and hosting migrations
Detect unauthorized DNS modifications

Example: Your domain’s A record changed overnight. Passive DNS tells you when, what it changed from, and what it changed to.

3. Investigation

Historical Research:

Incident response and forensics
Domain validation
Brand protection
Legal evidence gathering

Example: A phishing site existed for 3 days last year. Passive DNS provides the exact timeframe and infrastructure used.

How DETEQTIVE DNS Collects Passive DNS Data

DETEQTIVE DNS collects DNS data by:

Observing DNS queries and responses at strategic network points
Recording the domain name, record type, and data (IP, hostname, etc.)
Timestamping when records are first and last seen
Counting how many times each record is observed (sightings)

This creates a historical database of DNS records over time.

How the API Works

Two Search Directions

The API provides two complementary search methods:

Forward Search (RRSet)

Query: “Given a domain name, what are its DNS records?”

INPUT: Domain name (e.g., google.com)
OUTPUT: DNS records (A, AAAA, MX, etc.)

Use cases:

Find all IPs a domain has used
Discover subdomains
Track domain infrastructure
Find similar domains (typosquatting)

Reverse Search (RData)

Query: “Given an IP/CIDR/hostname, what domains point to it?”

INPUT: IP address, CIDR, or hostname
OUTPUT: DNS records containing that value in RDATA

Use cases:

Find all domains on a server
Map network infrastructure
Identify shared hosting
Track IP address reuse

Search Strategy

The API uses a streaming architecture:

Client sends query -> API server
Server queries the database
Results stream back as they’re found (NDJSON)
Client processes records incrementally
Status message sent when complete or on error

This enables:

Low latency - First results arrive quickly
Memory efficiency - No need to buffer entire result set
Large datasets - Handle millions of records
Early termination - Client can stop reading when satisfied

Search Strategies

Why Multiple Search Types?

Different questions require different search approaches. DETEQTIVE DNS provides five search types for maximum flexibility.

Exact Match (em)

Strategy: Find records matching exactly the specified domain.

SQL Equivalent: WHERE rrname = 'google.com'

When to use:

Looking up a specific, known domain
Checking if a domain exists in passive DNS
Getting precise records without noise

Performance: Fast - simple exact match index lookup

Example:

# Only google.com, not www.google.com or mail.google.com
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" https://api.deteqtive.com/api/v3/rrset/em/google.com/A

Right Match (rm)

Strategy: Find a domain and all its subdomains (right side wildcard).

SQL Equivalent: WHERE rrname = 'google.com' OR rrname LIKE '%.google.com'

When to use:

Infrastructure discovery
Subdomain enumeration
Security audits (finding all subdomains)
Mapping attack surface

Performance: Moderate - index range scan

Example:

# Returns: google.com, www.google.com, mail.google.com, api.google.com, etc.
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" https://api.deteqtive.com/api/v3/rrset/rm/google.com

Visual representation:

Search: google.com
Matches:
  |-- google.com
  |-- www.google.com
  |-- mail.google.com
  \_-- api.google.com
Does not match:
  \_-- notgoogle.com

Left Match (lm)

Strategy: Find a domain and all its parent domains (left side wildcard).

SQL Equivalent: WHERE 'www.google.com' LIKE rrname || '%'

When to use:

Understanding domain hierarchy
Finding zone apex
Traversing domain tree upward

Performance: Moderate to slow - less optimized than right match

Example:

# Search: www.google.com
# Returns: www.google.com, google.com, com
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" https://api.deteqtive.com/api/v3/rrset/lm/www.google.com

Visual representation:

Search: www.google.com
Matches:
  |-- www.google.com
  |-- google.com
  \_-- com
Does not match:
  |-- mail.google.com
  \_-- www.example.com

Fuzzy Match (fm)

Strategy: Find domains similar to search term using edit distance.

Algorithm: Levenshtein distance - counts insertions, deletions, and substitutions

When to use:

Typosquatting detection
Finding similar domains
Combating homograph attacks
Brand protection

Performance: Slow - requires comparison across many records

Parameters: fuzziness (1-3, default: 2) controls allowable distance

Example:

# Search: google.com with fuzziness=1
# Matches: google.com, googl.com, gooogle.com, g00gle.com
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" "https://api.deteqtive.com/api/v3/rrset/fm/google.com?fuzziness=1"

Edit distance examples:

google -> googl (1 deletion) = distance 1
google -> gogle (1 deletion) = distance 1
google -> g00gle (2 substitutions) = distance 2

Word Match (wm)

Strategy: Find domains containing the search term as a token/word.

Behavior: Token-based search - search term must appear as a word boundary

When to use:

Brand monitoring
Keyword searches
Finding related domains
Phishing campaign detection

Performance: Moderate - depends on search term popularity

Example:

# Search: paypal
# Matches: paypal.com, secure-paypal.com, paypal-verify.net
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" https://api.deteqtive.com/api/v3/rrset/wm/paypal

Word boundaries:

paypal in secure-paypal.com (word boundary)
paypal in mypaypalhelp.com (word boundary)
paypal in paypals.com might match depending on tokenization

NDJSON Streaming

Why NDJSON?

NDJSON (Newline-Delimited JSON) was chosen for several important reasons:

1. Streaming Support

Traditional JSON requires parsing the entire response:

{
  "results": [
    {"record": 1},
    {"record": 2}
  ]
}

You must wait for the closing } to know the JSON is complete.

NDJSON allows immediate processing:

{"record": 1}
{"record": 2}
{"status": "done"}

Each line is independently parseable.

2. Memory Efficiency

Traditional JSON:

Client must buffer entire response
Memory usage = size of full result set
Problem with large datasets (millions of records)

NDJSON:

Process one line at a time
Memory usage = size of single record
Handle unlimited dataset sizes

3. Progressive Processing

Example workflow:

for line in response.iter_lines():
    record = json.loads(line)
    if meets_criteria(record):
        process(record)
        if have_enough_results():
            break  # Stop reading, save bandwidth

Can stop reading at any point.

4. Error Recovery

If connection drops mid-response:

Traditional JSON: Entire response is invalid (incomplete)
NDJSON: All lines received so far are valid, usable data

Response Structure

Every response follows this pattern:

[DNS Record]        <- 0 or more data lines
[DNS Record]
[DNS Record]
...
[Status Message]    <- Always the last line (if complete)

Last line distinguishes success from error:

Success: {"eof": true, "count": N, "elapsed": 0.102, "error": null}
Error: {"eof": false, "count": N, "elapsed": 0.002, "error": "message"}
Truncated: If the eof status line is missing entirely, the response was truncated (connection lost, network issue, client timeout, etc.)

Important: Records are not sorted. The same query executed multiple times may return records in different order. This is by design for performance reasons - results are streamed as they’re found in the database.

Parsing Strategy

Python:

records = []
status = None

for line in response.iter_lines():
    data = json.loads(line)
    if 'eof' in data:
        status = data
    else:
        records.append(data)

if status is None:
    handle_truncation()  # Response was truncated
elif not status['eof']:
    handle_error(status['error'])

Bash/jq:

# Get only records
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" ... | jq -c 'select(.rrname)'

# Get only status
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" ... | tail -1 | jq -c

# Check if response was complete
if ! curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" ... | tail -1 | jq -e '.eof' > /dev/null 2>&1; then
    echo "Warning: Response was truncated"
fi

Error Handling Philosophy

User-Friendly Error Messages

The DETEQTIVE DNS API follows a principle of least information exposure for security:

Internal problems — a query timeout, a transient connectivity issue, and so on — are never surfaced verbatim. They are mapped to a small set of normalized, user-facing messages that describe what happened without exposing how the service works:

{"eof":false,"count":0,"elapsed":0.002,"error":"query timeout - please refine your search or try again later"}
{"eof":false,"count":0,"elapsed":0.002,"error":"a temporary error occurred - please try again later"}

Error Categories

The API normalizes errors into four categories:

1. Query Timeout

Message: “query timeout - please refine your search or try again later”
Cause: Query exceeded 120 seconds
Solution: Add filters, reduce scope

2. Temporary Error

Message: “a temporary error occurred - please try again later”
Cause: Temporary problem on our side
Solution: Retry

3. Invalid Request

Message: “the request could not be completed - please check your search parameters”
Cause: Invalid query structure
Solution: Check parameters

4. Generic Error

Message: “an error occurred while processing your request”
Cause: Unexpected error
Solution: Contact support

Partial Results

Errors may occur after some results are returned:

{"rrname":"example.com","rrtype":"A","rdata":"93.184.216.34",...}
{"rrname":"example.com","rrtype":"A","rdata":"93.184.216.35",...}
{"eof":false,"count":2,"elapsed":0.002,"error":"query timeout - please refine your search or try again later"}

The count field shows how many records were successfully returned before the error.

Time-Based Data

Why Timestamps Matter

Passive DNS without timestamps is just a list. Timestamps provide:

Chronology - When things changed
Freshness - What’s current vs. historical
Duration - How long something existed
Correlation - Match events across systems

First Seen vs. Last Seen

first_seen:

When this exact record was first observed
Indicates when infrastructure was established
Useful for finding newly created domains

last_seen:

When this exact record was last observed
Indicates if record is still active
Useful for finding expired/changed records

Example:

{
  "rrname": "example.com",
  "rrtype": "A",
  "rdata": "93.184.216.34",
  "first_seen": "2020-01-01T00:00:00Z",
  "last_seen": "2024-01-01T00:00:00Z",
  "sightings": 50000
}

Interpretation:

Record active for ~4 years
Observed 50,000 times
Last seen recently (likely still active)

Sightings

Definition: Number of times this exact record was observed.

High sightings:

Popular/important domain
Frequently queried
Stable infrastructure

Low sightings:

Rare/obscure domain
Short-lived
Newly created

Use case: Filter noise by requiring minimum sightings.

Unix Timestamps

API uses Unix timestamps (seconds since 1970-01-01 00:00:00 UTC) for filters.

Default Time Window:

last_seen_after: Defaults to 14 days ago
last_seen_before: Defaults to now
first_seen_after: No default (unlimited)
first_seen_before: No default (unlimited)

Important: Queries return only records seen in the last 14 days unless you override last_seen_after. To query all historical data, set last_seen_after=0.

Convert human date to Unix:

# Linux
date -u -d '2024-01-01' +%s
# Output: 1704067200

# macOS
date -u -j -f '%Y-%m-%d' '2024-01-01' +%s

Convert Unix to human date:

# Linux
date -u -d '@1704067200'
# Output: Mon Jan  1 00:00:00 UTC 2024

# macOS
date -u -r 1704067200

Time Filtering Strategies

Query all historical data:

# Override default 14-day window
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" "https://...?last_seen_after=0"

Find new infrastructure:

# Records first seen in last 7 days
SEVEN_DAYS_AGO=$(date -u -d '7 days ago' +%s)
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" "https://...?first_seen_after=$SEVEN_DAYS_AGO"

Find likely expired records:

# Records last seen more than 1 year ago
ONE_YEAR_AGO=$(date -u -d '1 year ago' +%s)
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" "https://...?last_seen_before=$ONE_YEAR_AGO"

Find active records:

# Records seen in last 30 days
THIRTY_DAYS=$(date -u -d '30 days ago' +%s)
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" "https://...?last_seen_after=$THIRTY_DAYS"

Bailiwick Concept

What is Bailiwick?

Bailiwick is the DNS zone where a record was authoritatively observed.

Simple explanation: The bailiwick is the “zone of authority” for a DNS record.

Example

{
  "bailiwick": "google.com",
  "rrname": "www.google.com",
  "rrtype": "A",
  "rdata": "142.250.185.46"
}

Interpretation:

Record name: www.google.com
Bailiwick: google.com
Meaning: This A record was observed in the google.com zone

Why Bailiwick Matters

1. Authority Verification

Know whether a record came from:

The authoritative zone
A delegated zone
A cached/recursive resolver

2. Zone Correlation

Group records by authoritative zone:

Bailiwick: example.com
  |-- example.com A 1.2.3.4
  |-- www.example.com A 1.2.3.5
  \_-- mail.example.com A 1.2.3.6

3. Delegation Detection

Identify zone delegations:

rrname: subdomain.example.com
bailiwick: subdomain.example.com  <- Delegated zone

vs.

rrname: subdomain.example.com
bailiwick: example.com  <- Parent zone

When to Use Bailiwick

Include bailiwick when:

Analyzing zone structure
Verifying authority
Correlating related records
Understanding delegations

Skip bailiwick when:

Only need IP addresses
Minimizing bandwidth
Bailiwick not relevant to analysis

How to Request Bailiwick

Via query parameter:

curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" "https://api.deteqtive.com/api/v3/rrset/em/google.com?withbailiwick"

Via path parameter:

curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" https://api.deteqtive.com/api/v3/rrset/em/www.google.com/A/google.com

Path parameter also filters by bailiwick.

Performance Considerations

Query Performance Factors

1. Search Type

Performance ranking (fast to slow):

Exact Match (em) - Direct index lookup
Right Match (rm) - Index range scan
Left Match (lm) - Less optimized scan
Word Match (wm) - Token search
Fuzzy Match (fm) - Expensive distance calculation

2. Filters

Impact on performance:

Record type filter (/A) - Significant improvement
Time filters - Moderate improvement
Limit - Memory and transfer improvement
Bailiwick - Depends on use case

3. Result Size

Small result sets (<1000):

Fast query
Quick transfer
Low memory

Large result sets (>10000):

Longer query time
More bandwidth
Consider pagination

4. Result Ordering

Results are NOT sorted - this is intentional for performance:

Why unsorted?

Performance: Sorting large datasets (millions of records) is expensive
Streaming: Records are returned as found, enabling low latency
Scalability: No need to buffer entire result set before sending
Distributed queries: Data may come from multiple shards

Impact:

Same query may return records in different order
No guarantee of consistent ordering between requests
Records arrive in database scan order (non-deterministic)

Solution: If you need consistent ordering, sort results client-side:

# Sort by IP
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" ... | jq -s 'map(select(.rrname)) | sort_by(.rdata)'

# Sort by timestamp
curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" ... | jq -s 'map(select(.rrname)) | sort_by(.first_seen)'

Storage Architecture

DETEQTIVE DNS is backed by a columnar, analytics-optimized data store:

Why it’s fast:

Columnar storage - Efficient compression
Fast aggregations - Quick counting, filtering
Parallel processing - Multi-core queries
Large datasets - Billions of records

Performance characteristics:

Excellent for analytical queries
Optimized for read-heavy workloads
Efficient time-series data
Fast pattern matching

Timeout Strategy

120-second timeout prevents:

Runaway queries
Resource exhaustion
Poor user experience

Default time window:

Queries return only records seen in the last 14 days
This prevents accidentally querying billions of historical records
Override with last_seen_after=0 to query all data

If hitting timeouts:

Add limit parameter
Filter by time range
Use specific record type
Choose narrower search type
Add bailiwick filter

Security and Privacy

Data Sanitization

What the API does NOT expose:

Raw query logs
Source IPs of DNS queries
Personal information
Sensitive infrastructure details
Backend system information

What the API DOES provide:

Aggregated DNS records
Time ranges (not exact timestamps of each query)
Sighting counts (not query sources)
Public DNS information only

Error Message Security

Error messages are sanitized to prevent information disclosure:

Bad (don’t do this):

{"error": "<internal driver error / host:port / stack trace>"}

Good (what DETEQTIVE DNS does):

{"error": "a temporary error occurred - please try again later"}

Rate Limiting

Implementation-specific: Consult your administrator for rate limits.

Purpose:

Prevent abuse
Ensure fair access
Protect infrastructure

Authentication

Implementation-specific: May require API keys.

If enabled:

curl -H "Authorization: Bearer $DETEQTIVE_TOKEN" ...

Data Collection and Quality

Collection Methods

Passive DNS data comes from:

Recursive DNS servers observing queries/responses
Authoritative servers logging queries
Network sensors monitoring DNS traffic
Zone transfers (where permitted)

Data Quality Factors

Completeness:

Not all DNS queries are observed
Coverage depends on sensor placement
Popular domains have better coverage

Accuracy:

Records reflect actual observations
Timestamps show observation time, not TTL
Sighting counts are approximate

Freshness:

Data updated continuously
Last_seen reflects most recent observation
Gaps possible if domain not queried

Limitations

What Passive DNS CAN’T tell you:

Current state - Only historical observations
All records - Only observed records
Exact TTLs - Not captured
Query sources - Privacy preserved
Complete coverage - Depends on sensors

What Passive DNS CAN tell you:

Historical IPs - What IPs were used
Time ranges - When records were active
Popularity - Via sighting counts
Infrastructure - Domain relationships
Changes - When DNS changed

Summary

The DETEQTIVE DNS API is designed with:

Usability - Clear, consistent interface
Performance - Streaming, optimized queries
Security - Sanitized errors, privacy protection
Flexibility - Multiple search strategies
Reliability - Timeout protection, error recovery

Key principles:

Stream results for efficiency
Provide time context for all data
Avoid exposing internal implementation details
Support diverse search strategies
Enable large-scale analysis

Ready to use the API? Start with the Tutorial Need specific examples? See How-To Guides Looking for details? Check Reference

← Previous Reference

Explanation

Explanation: Understanding DETEQTIVE DNS API

Authentication

Table of Contents

What is Passive DNS?

The Concept

Why Passive DNS Matters

1. Security Research

2. Infrastructure Monitoring

3. Investigation

How DETEQTIVE DNS Collects Passive DNS Data

How the API Works

Two Search Directions

Forward Search (RRSet)

Reverse Search (RData)

Search Strategy

Search Strategies

Why Multiple Search Types?

Exact Match (em)

Right Match (rm)

Left Match (lm)

Fuzzy Match (fm)

Word Match (wm)

NDJSON Streaming

Why NDJSON?

1. Streaming Support

2. Memory Efficiency

3. Progressive Processing

4. Error Recovery

Response Structure

Parsing Strategy

Error Handling Philosophy

User-Friendly Error Messages

Error Categories

Partial Results

Time-Based Data

Why Timestamps Matter

First Seen vs. Last Seen

Sightings

Unix Timestamps

Time Filtering Strategies

Bailiwick Concept

What is Bailiwick?

Example

Why Bailiwick Matters

1. Authority Verification

2. Zone Correlation

3. Delegation Detection

When to Use Bailiwick

How to Request Bailiwick

Performance Considerations

Query Performance Factors

1. Search Type

2. Filters

3. Result Size

4. Result Ordering

Storage Architecture

Timeout Strategy

Security and Privacy

Data Sanitization

Error Message Security

Rate Limiting

Authentication

Data Collection and Quality

Collection Methods

Data Quality Factors

Limitations

Summary