fix(app): collection fetches addresses with Korean paths and words
The request line is ASCII: other characters and spaces in the path and query go as UTF-8 %HH, while existing %HH and separators stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -135,6 +135,10 @@ def _connect(ips: list, port: int, deadline: float) -> socket.socket:
|
||||
raise
|
||||
|
||||
|
||||
def _ascii(part: str) -> str:
|
||||
return urllib.parse.quote(part, safe="/%:@!$&'()*+,;=?[]~")
|
||||
|
||||
|
||||
def _get(url: str, deadline: float, dog: _Watchdog):
|
||||
"""One hop: ("redirect", next url) or ("body", bytes)."""
|
||||
import http.client
|
||||
@@ -161,7 +165,9 @@ def _get(url: str, deadline: float, dog: _Watchdog):
|
||||
else:
|
||||
conn = http.client.HTTPConnection(p.hostname, port)
|
||||
conn.sock = sock
|
||||
path = (p.path or "/") + (f"?{p.query}" if p.query else "")
|
||||
# The request line is ASCII: other characters (and spaces) go as UTF-8 %HH; existing %HH
|
||||
# and the reserved separators stay as they are.
|
||||
path = _ascii(p.path or "/") + (f"?{_ascii(p.query)}" if p.query else "")
|
||||
conn.request("GET", path, headers={"User-Agent": USER_AGENT, "Accept-Encoding": "identity"})
|
||||
r = conn.getresponse()
|
||||
if r.status in (301, 302, 303, 307, 308):
|
||||
|
||||
@@ -338,7 +338,8 @@ still deleted (below).
|
||||
local network. Every redirect is checked the same way. At most 1 MB and 15 s in all: the name lookup,
|
||||
connecting, TLS, the headers and the body of every redirect count, and a server that sends
|
||||
slowly, even byte by byte within the per-read limit, is cut off at 15 s. A response cut short is a
|
||||
failure. The check logs nothing about the address. The collected text is web
|
||||
failure. Characters outside ASCII (and spaces) in the path and words go as UTF-8 `%HH`; existing
|
||||
`%HH` and separators stay. The check logs nothing about the address. The collected text is web
|
||||
content: it goes only to `<dir>/<name>/<YYYY-MM-DD>.jsonl` (one JSON object per line: `at`, then
|
||||
`title`, `url`, `text` as the kind has them), never to the brain or a log.
|
||||
- Day files older than `keepDays` are deleted in every folder of `dir` at each check, also of
|
||||
|
||||
Reference in New Issue
Block a user