Very interesting piece and I think San Francisco isn't alone in the need for some form of greater regional governance. My mind immediately jumps to all the regional planning bodies throughout the U.S., like CMAP (Chicago Metropolitan Agency for Planning) in Illinois, which covers an area vastly greater than the City of Chicago but whose powers are purely advisory. From what I understand, they put out excellent regional plans full of great TOD and YIMBY proposals. Yet it's ultimately the Aldermen of Chicago who exert a kind of feudal authority over every little land-use decision in their patches of turf. With a playing field like this, it's no wonder NIMBYs dominate (or why Aldermen keep going to jail, for that matter). Consolidating land-use authority in a regional body certainly would tilt the balance of power towards pro-housing forces.
But the transit issue is where I think your case is strongest and most generalizable. I live in DC, where our regional metro system WMATA is chronically under-funded and terrified of making long-term investments needed to ensure the decent and reliable service a metro area of our size deserves. WMATA has to go hat-in-hand to Maryland and Virginia soliciting voluntary contributions for a system that reaches very deeply into both states. The share of operating costs recovered by fares is pretty low, but I don't think it's totally out of the ordinary for U.S. transit systems. DC residents end up bearing a much greater cost for what is a truly regional system. But we simply can't compel Maryland and Virginia to pay their fare share. It sounds like BART has the same issue? We could use some regional bodies with actual taxing authority whose jurisdictions actually align with the reach of our infrastructure needs. Maybe "save our metro" is a useful political rallying cry for this.
Are there any examples of cities around the world you've seen that do this particularly well? Greater London, essentially coterminous with the Tube, comes to mind. But I'd love to do some research on further case studies, perhaps for one of my ROP BBI pieces.