Yeah I’m understanding that too. Which shouldn’t be a surprise really. As I know full well that how one asks a question massively influences the answer you get. How do I know… well I’ve been married a long time. 😂. Actually more seriously one thing I’m pondering at the moment is exactly this dilemma. My buddy and I wrote a book 3 years back, called how to build a data driven culture (it was really a book, about Change, using data as its example). We’ve an idea to turn it around and write one called - nobody wants a data driven culture. Our idea is get get an AI to read our book and “produce” this new version. For us to edit of course. And we’d like it to be a lot shorter than the original 250 pages. So, do we ask AI to summarise the first book into say 50 pages first. Then get it to turn the arguments on their head. But that might miss out key things. What questions do we ask it to draw the content out we want. Questions, questions questions.
There are a few ways to do this depending on how concerned you are about being faithful to the ideas of your original book. If your priority is trimming it down to 50 pages and you’re willing to sacrifice some of your points or the rhetoric leading up to conclusions, you can dump the full book into a frontier model and ask it to summarize it.
If you want to optimize for quality, you could go a chapter at a time or even a thousand words at a time but you need to do more than simply ask it to summarize the text. Ask it to summarize the thesis of that chapter, important examples, supporting arguments, the logical progression from one point to another… things like that. This forces the AI to look at what you’ve written, determine which parts are important, and to preserve those parts. Don’t place a page limit on these summaries. Just have it do them with the goal of creating a workable summary text. Do this for every chapter. After you’ve finished, feed that collection of all of the summaries back into the AI and ask it then to summarize that down to your goal of 50 pages.
You should be reading each section on your own to make sure that it is actually capturing the spirit and presentation of your arguments. Then read the new summaries as well. After you’ve finished reducing it to the desired page count, you could run a verification check where you give it the original text and the summarized text and ask it to check for lost arguments and things like that, but if you’ve been reading it as it produced each chapter summary you’ll already have caught any mistakes.
That’s how I would approach something like that. The big thing is to break it up into chunks and to tell the AI exactly what you want it to do at each phase of the process and then verify it yourself. Too many people just blindly trust the AI and sometimes that can work, but sometimes it can lead to bad outcomes.
John you are doing the real work for us on X - I ran screaming from that platform a few years ago so have a emoji 🙏
On your poll - you made me lie because I've gone through all the stages of use and while I do use it a lot for tech stuff it's certainly not my only use case.
On Ward and Wire - I love them and I love that you drew them and I am jealous of your talents.
On deterministic systems (most everything I ever did in my career) versus probabilistic AI - its pretty screwy and you did a good job explaining how these nuances hit prompting. I don't know if you saw that NYT article where one coders explicitly put in one of his skill files that you MUST pass the python unit tests before you commit otherwise it's EMBARASSING!
I built a audit bot for my High School kids to check their code against the AP test requirements and one of them complained that they had met the requirement for a complex function that the AI was flagging and I told them to push back on the AI and show them the code and they were flabbergasted that it would change its mind.
Thanks for the feedback. I tried to simplify things a bit more in this issue after getting some feedback that the first issue was a bit too technical in places. Finding that sweet spot between being technical enough for readers like you and accessible enough for people who are new to the field is challenging, but when it works it feels like magic.
I love the story about your students learning how they can pushback against AI.
Yeah I’m understanding that too. Which shouldn’t be a surprise really. As I know full well that how one asks a question massively influences the answer you get. How do I know… well I’ve been married a long time. 😂. Actually more seriously one thing I’m pondering at the moment is exactly this dilemma. My buddy and I wrote a book 3 years back, called how to build a data driven culture (it was really a book, about Change, using data as its example). We’ve an idea to turn it around and write one called - nobody wants a data driven culture. Our idea is get get an AI to read our book and “produce” this new version. For us to edit of course. And we’d like it to be a lot shorter than the original 250 pages. So, do we ask AI to summarise the first book into say 50 pages first. Then get it to turn the arguments on their head. But that might miss out key things. What questions do we ask it to draw the content out we want. Questions, questions questions.
There are a few ways to do this depending on how concerned you are about being faithful to the ideas of your original book. If your priority is trimming it down to 50 pages and you’re willing to sacrifice some of your points or the rhetoric leading up to conclusions, you can dump the full book into a frontier model and ask it to summarize it.
If you want to optimize for quality, you could go a chapter at a time or even a thousand words at a time but you need to do more than simply ask it to summarize the text. Ask it to summarize the thesis of that chapter, important examples, supporting arguments, the logical progression from one point to another… things like that. This forces the AI to look at what you’ve written, determine which parts are important, and to preserve those parts. Don’t place a page limit on these summaries. Just have it do them with the goal of creating a workable summary text. Do this for every chapter. After you’ve finished, feed that collection of all of the summaries back into the AI and ask it then to summarize that down to your goal of 50 pages.
You should be reading each section on your own to make sure that it is actually capturing the spirit and presentation of your arguments. Then read the new summaries as well. After you’ve finished reducing it to the desired page count, you could run a verification check where you give it the original text and the summarized text and ask it to check for lost arguments and things like that, but if you’ve been reading it as it produced each chapter summary you’ll already have caught any mistakes.
That’s how I would approach something like that. The big thing is to break it up into chunks and to tell the AI exactly what you want it to do at each phase of the process and then verify it yourself. Too many people just blindly trust the AI and sometimes that can work, but sometimes it can lead to bad outcomes.
That great John. Thanks.
Another good’n. Thanks, John 👌
Thanks for reading it. I’m glad you’re enjoying it.
I literally laughed out loud twice, reading the humor section. Co-Pilot... why!?
John you are doing the real work for us on X - I ran screaming from that platform a few years ago so have a emoji 🙏
On your poll - you made me lie because I've gone through all the stages of use and while I do use it a lot for tech stuff it's certainly not my only use case.
On Ward and Wire - I love them and I love that you drew them and I am jealous of your talents.
On deterministic systems (most everything I ever did in my career) versus probabilistic AI - its pretty screwy and you did a good job explaining how these nuances hit prompting. I don't know if you saw that NYT article where one coders explicitly put in one of his skill files that you MUST pass the python unit tests before you commit otherwise it's EMBARASSING!
I built a audit bot for my High School kids to check their code against the AP test requirements and one of them complained that they had met the requirement for a complex function that the AI was flagging and I told them to push back on the AI and show them the code and they were flabbergasted that it would change its mind.
Thanks for the feedback. I tried to simplify things a bit more in this issue after getting some feedback that the first issue was a bit too technical in places. Finding that sweet spot between being technical enough for readers like you and accessible enough for people who are new to the field is challenging, but when it works it feels like magic.
I love the story about your students learning how they can pushback against AI.
I’m having fun with AI in the classroom. I wouldn’t have taken on this class if I didn’t know AI had my back.