09:41 <herrwolfe45> I'm working on ticket #3696 and am a bit stuck on what type of documentation I should write - I'm thinking I should write a how-to describing how one should install twisted and it's various optional dependencies. This how-to would be in the narrative docs for twisted core. Does this sound like a decent idea?
09:41 <glyph> herrwolfe45: That sounds fantastic.
09:42 <herrwolfe45> glyph: excellent, I'll start with that - thanks
09:45 <herrwolfe45> ergh its
09:48 <exarkun> When writing that, don't forget that `pip install Twisted` is the least preferred installation method.  When possible people should prefer an OS-supplied package.  (But it's certainly not always possible.)
09:49 <exarkun> (Or possibly tailoring different sections of the document to different audiences makes sense - developers working on Twisted have slightly different requirements from developers using Twisted have very different requirements from end-users using an application that depends on Twisted.)
09:50 <glyph> exarkun: Least preferred by whom?
09:51 <glyph> exarkun: I would actually say "pip install Twisted" is the _most_ preferred installation method at this point.  OS-provided packages have kinda dropped the ball at this point as far as keeping up with releases and providing isolated, repeatable installations.
09:52 <glyph> The major reasons to prefer OS-vendor-provided packages are integrations with things like GTK and pyObjC, which are definitely not the most popular use-case :-\
09:52 <exarkun> It's most preferred if it's most preferred, I agree - but that's also sort of a useless statement. ;)
09:52 <exarkun> If your OS vendor has packaged the version of Twisted you need, you should use that package.
09:52 <exarkun> That's why I said "when possible".
09:53 <glyph> exarkun: Why?
09:53 <exarkun> Because it's going to get security updates, for example.
09:53 <glyph> exarkun: We ship security updates via pip as well, though.
09:54 <exarkun> Yes, but the upgrade experience for pip isn't as good.
09:54 <exarkun> You want unattended-upgrades to just install the security update - if possible.
09:54 <exarkun> You want as few installations on a system as possible.  You don't want to have to remember to pip upgrade all of your deployments.
09:54 <glyph> exarkun: Except unattended-upgrades is insufficient, because you need something that is going to restart your Twisted services for you when the updates are available; and your OS isn't going to help with that anyway.
09:55 <exarkun> And (not security related) it's not going to screw with other OS-provided packages that have a Twisted dependency.  Perhaps those apps break when you install the newer Twisted into your environment (so, yay, virtualenv or whatever, but `pip` doesn't /require/ virtualenv).
09:55 <glyph> exarkun: also, like, Docker completely breaks unattended-upgrades anyway, which I am still trying to figure out how to deal with :-)
09:55 <glyph> Aah.
09:55 <exarkun> glyph: You're basically saying "OS packages aren't 100% perfect so use pip"
09:55 <glyph> Right.
09:55 <exarkun> ignoring the fact that pip is also not 100% perfect
09:56 <glyph> exarkun: Virtualenv is the thing, really, not pip.
09:56 <exarkun> There are advantages and disadvantages to both.
09:57 <exarkun> Maybe you're even right that on balance virtualenv & pip gives you the best trade-offs - but only if you understand how to manage it.  I'd argue that someone who understands those things deeply isn't going to read our installation documentation anyway: they already know how to do this.
09:57 <exarkun> For the folks most likely to need this documentation (again, *if possible*) their OS packages are going to provide a better experience.
09:58 <exarkun> And I'm not saying don't provide docs about both approaches, I'm just saying don't provide docs that unconditionally say "install Twisted with pip no matter what".
09:58 <glyph> exarkun: OK, let's talk about the folks most likely to need this documentation.
09:58 <glyph> My understanding of the target audience is that this is people with a cursory understanding of Python but no real understanding of packaging or the tradeoffs involved.
09:59 <exarkun> I gave three example target audiences above
09:59 <exarkun> I think your description is pretty close to the third example and it's probably the group most in need of any kind of installation docs, yes.
10:00 <glyph> exarkun: My experience of that audience is that they know what "pip" is because they have 2 other dependencies they've already installed, both of which had installation documentation that told them to do "sudo pip install".
10:00 <exarkun> (But the other groups are interested in the particulars of how they might get all of Twisted's dependencies since Python's packaging ecosystem is still immature that you can't *totally* rely on that going smoothly)
10:01 <glyph> exarkun: Frequently they've also installed 2 or 3 versions of Python by downloading source code or installers from python.org.
10:01 <exarkun> You're narrowing in on the *really* screwed end of the spectrum.
10:02 <exarkun> Our docs can't be a comprehensive course in how not to screw up your computer, either.
10:02  → iffy (was magmatt) joined
10:02 <glyph> exarkun: Yes, and I don't want them to be.
10:03 <glyph> exarkun: But, if we say "use your operating-system provided packages", this presupposes a level of knowledge of their operating environment that I think literally zero of the people in this audience (people who need some kind of set-up/installation documentation) actually have.
10:03 <exarkun> Again, I don't think that's *all* we should say.
10:04 <glyph> exarkun: I'm not arguing that we should just say "use pip, hth hand" either.
10:05 <glyph> In fact I'm not sure that I'm arguing *for* anything in particular.  I think our preference for saying "use your OS installed packages" produces undesirable outcomes and I would like to explore what we should be saying instead.
10:06 <exarkun> Okay
10:06 <glyph> Maybe we should have an entire document that says "using the Twisted packages from your operating system"
10:07 <glyph> The chief undesirable outcome I have noticed this guidance producing is people having really no idea what "operating system packages" are, and having to deal with multiple environments (almost always: OS X for dev, Linux for prod)
10:07 <glyph> so they build some knowledge in one environment that completely fails to transfer to the other
10:08 ⇐ itamarst quit (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) Quit: Leaving.
10:08 <exarkun> That seems like the kind of basic misunderstanding that it might be very difficult for us to correct.
10:08 <glyph> exarkun: well, this is the advantage of suggesting a virtualenv-based approach to installations.
10:08 <exarkun> For one thing, someone who thinks they know about installing software because they've done it on OS X might not even think to look at our documentation to learn how to install things on Linux.
10:09 <exarkun> glyph: Calling it an advantage seems a little tricky to me.
10:09 <exarkun> It's definitely a trait.
10:10 <exarkun> Offering one lowest-common-denominator solution is good because once someone learns it they don't have to learn something else but it's bad because maybe in learning something else they could produce a better result.
10:12 <glyph> I'm not really sold on the OS-pavkages route being "a better result"
10:13 <glyph> Thus far, the advantages we've discussed are: half of an automated security updates solution if you're using Debian, and reduced build interdependencies for desktop / GUI integration stuff
10:15 <glyph> The advantages of virtualenv+pip are: portability, a greater degree of environment isolation and therefore predictability, and more knowledge sharing with other projects (since most other python projects recommend that deployment strategy)
10:15 ⇐ khorn quit (~funsize@pool-72-64-101-86.dllstx.fios.verizon.net) Read error: Connection reset by peer
10:16 <exarkun> Another factor worth mentioning, probably, is chances of actually producing a working installation
10:17 <exarkun> virtualenv & pip have extra dependencies - eg, a C compiler and various headers
10:17 <exarkun> If you actually want `pip install Twisted` to work in a virtualenv, better install libssl-dev (or is it libssl-devel) and other such things first
10:18 <exarkun> (using your OS's package manager!)
10:19 <glyph> Good point, the portability argument is weakened considerably by the totally non-portable nature of development dependencies
10:20 <Alex_Gaynor> Isn't it a goal that twisted be installable without a C compiler?
10:20 <glyph> Not just libssl-dev, of course: don't forget libffi-dev ;)
10:20 <exarkun> Alex_Gaynor: Yes - but that's just Twisted. If you're installing Twisted and all of its dependencies, which you are if you use pip in a virtualenv, life is a lot harder.
10:20 <glyph> Alex_Gaynor: Sure, but then you won't get SSL support (unless somehow cryptography is going to be installable without a C compiler too)?
10:21 <exarkun> Right, so even in the best possible case, you end up with a crippled Twisted install - which may not be what you wanted.
10:21 → khorn joined (~funsize@pool-72-64-101-86.dllstx.fios.verizon.net)
10:24 <exarkun> So, getting practical, would our virtualenv & pip installation documentation include a section on how to use various OS's native package managers to install those dependencies?
10:25 <glyph> exarkun: I think so.
10:25 <hynek> that’s three lines to cover Linux & FreeBSE. OS X is the real issue because it’s a) unstandardized and b) used by a lot of junior peole
10:26 <glyph> hynek: "type cc in a terminal and follow the on-screen instructions"
10:26 <glyph> hynek: we _don't_ need to help you install PyGame and GTK development libraries, remember: libffi and openssl are both bundled with the platform
10:27 <exarkun> hynek: But once you've got those three lines, you've brought into the extra conceptual overhead - and putting a "python-twisted" package name somewhere on those lines doesn't increase that burden (and you get to stop there instead of talking about virtualenv & pip if you want)
10:27 <exarkun> So *some* of the portability advantage has been lost
10:27 <glyph> exarkun: except you _might_ not get to stop there, because your *other* dependencies, which you want to integrate with Twisted, may not install cleanly in that environment (or may require the use of 'sudo pip')
10:28 <exarkun> glyph: Sure, but some people will get to stop there.
10:28 <exarkun> This suggests to me the docs should be something like
10:29 <hynek> once people start using python-twisted, they start installing crap into their site-packages and then they come crying
10:29 <glyph> hynek: let's be specific :-).  What does "come crying" mean?  What breaks?
10:30 <hynek> glyph: https://hynek.me/articles/virtualenv-lives/ the examples are extra-twistedish but the points remain
10:30 <exarkun> "Maybe this OS-specific command is sufficient for you: <dpkg / yum / whatever to install Twisted>.  If you need a newer version, start with this OS-specific command: <dpkg / yum / whatever to install Twisted's dependencies> and then <virtualenv & pip instructions instructions>"
10:30 → itamarst joined (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net)
10:30 <exarkun> hynek: People will /have/ to install some native packages.
10:30 <hynek> it’s not just about newer versions. it’s also about klein, treq etc pp
10:30 ⇐ trenton42 quit (~trenton@199.116.53.69) Ping timeout: 240 seconds
10:31 <hynek> exarkun: that statement has absolutely nothing to do with my point. not sure whether it’s my fault or yours though. :)
10:31 <exarkun> hynek: Given that, I don't think you can get away from your concern about people incorrectly installing unpackaged software in system paths.
10:32 <glyph> exarkun: I think there's an unspoken assumption here: the number of native packages required from the platform is small (literally 2: openssl-dev and libffi-dev); the number of python packages required by application developers is large (klein treq characteristic txamqp txkazoo ampoule service_identity pyasn1_modules and so on and so on)
10:33 <exarkun> Or maybe not "If you need a newer version" but "If you'd prefer to have an isolated installation (for example, so you can have multiple versions of Twisted) ..."
10:33 <exarkun> glyph: It's larger than that.
10:33 <exarkun> glyph: And even if it's just one thing, it's the conceptual overhead of switching to another tool.
10:33 <glyph> exarkun: Perhaps I should have said "typically"; for some developers it's huge. What am I forgetting, though?
10:33 <exarkun> The number of arguments you copy/paste along with that tool's invocation command is sort of irrelevant.
New messages since you tabbed out
10:34 <glyph> exarkun: On OS X and Windows, you don't actually need to switch to another tool though
10:34 <glyph> exarkun: on Windows, cryptography wheels address the need for openssl and cffi wheels address the need for libffi
10:34 ⇐ itamarst quit (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net) Client Quit
10:34 <dstufft> glyph: also python-dev
10:34 <glyph> exarkun: on OS X, "install the developer tools" is a single operation you do once and almost _certainly_ need to do to get a lot of things done
10:35 <exarkun> Are there binary wheels for cryptography now?
10:35 <hynek> I think jp’s point is basically that there should be as little hurdles as possible to try Twisted which I completely agree with.  Although if documented, it should be pointed out that it’s just a sneak-peak and if pursuited for anything else than writing distribution-specific software, it will end in tears.
10:35 <glyph> exarkun: for windows, yes!
10:35 <glyph> exarkun: it is ~~~amazing~~~
10:36 <exarkun> Okay, cool.  I didn't notice when that happened.
10:36 <glyph> exarkun: they statically link openssl, you don't need to go to that shitty website any more
10:36 <hynek> it’s still impossible to build linux binary wheels :(
10:36 <exarkun> Right.  But you do need to pip upgrade every time an OpenSSL security update comes out (~ once a month)
10:36 <glyph> exarkun: Yup.
10:36 <exarkun> But it's Windows.  So too bad, that's what you have to do.  We don't have to argue about that.
10:37 <glyph> exarkun: Or it's Docker and that's what you have to do too ;-) (seriously somewhere else we should have a conversation about container security updates because I do not know what the _heck_ is going on with that)
10:37 → itamarst joined (~itamarst@pool-108-20-253-103.bstnma.east.verizon.net)
10:38 <exarkun> Okay I gotta do some work.
10:38 <hynek> glyph: there is none. my tour into docker made me drink.
10:38 <glyph> exarkun: I probably should too.
10:38 — glyph wonders if he can capture some notes from this conversation in a useful way