Through 6/18/24

Video/audio transcript

00:0000:00 - hey guys so right now I’m going to be making the ministry progress summary video here through June 18th
00:0700:07 - 2024 so it’s been a couple months here um I’ll be going through a good bit of the programming things that I got done
00:1400:14 - here so I have continued to chip away at the automation that I want to underpin all of the content production in the
00:2200:22 - long term so we’ll be talking about some of the to-dos I left for myself in the last Ministry progress summary and then
00:2900:29 - I’ll go through kind of how I addressed a couple of those as well as you know this third one here with the transcripts
00:3400:34 - which was uh not so much on the list but I I added it because it was automation as well and then kind of what the upshot
00:4200:42 - of all of this is in terms of video production and uh what it means in terms of content that will be posted here in
00:4900:49 - the next couple weeks and then finally as per normal we’ll close with what my plans are for the next little bit
01:0001:00 - all right so before we get to the specifics of what I did this time I’m just going to briefly recap some of what
01:0601:06 - my short-term goals were at the end of the last Ministry progress update so one of the first things that I wanted to get
01:1301:13 - done uh moving forward at the end of the last one was setting up the automatic silence removal to work in my automated
01:2301:23 - command line video processing project so up until now I have been manually processing ing all of the videos that I
01:3201:32 - upload using a graphical user interface it’s a it’s a web app that lets you trim videos to automatically remove the
01:3901:39 - silent parts and it does basically the same thing that my command line tool does and the reason why I was having to
01:4401:44 - do that was because my commandline tool was not playing nice with the recordings from Zoom so that was one of the things
01:5101:51 - that I wanted to fix and then I also wanted to set up further automation with the topic transitions in my videos so
01:5901:59 - these are kind of the black screens the last several seconds that have the title for each section uh just giving a nice
02:0602:06 - transition between the parts of the videos and I wanted to automatically generate these video segments rather
02:1302:13 - than having to manually record them all and then standardize the length which is what I’ve been doing up until now I knew
02:2002:20 - it was possible for me to automatically generate these video segments so that’s what I wanted to set up and then finally
02:2602:26 - I wanted to set up a podcast for all of the content as well so thus far all of the video content has been going up on
02:3502:35 - YouTube obviously the web pag is on the website um but to support audio uh so this would be like on Spotify and apple
02:4202:42 - podcast which are as far as I know the two most dominant podcasting platforms at the moment um I need to do some work
02:4902:49 - to get the content ready to go up on the podcast platforms so these were the three things that I had sort of set off
02:5802:58 - to work on in this last little bit and I’m going to go over what progress I made on these as well as the other
03:0503:05 - things that I did that weren’t so much on this list but were the other places where I spent my
03:1303:13 - time so first off going to cover the automated silence removal sort of task that I left for myself and what progress
03:2003:20 - I made on that so it turns out that there is a sort of obscure recording setting on Zoom that I hadn’t come
03:2703:27 - across before at the time that was trying to figure out how to make my video processing workflow work with the
03:3403:34 - zoom recordings there’s this setting called optimize for third-party video editor and this is on the recordings
03:4103:41 - part of the settings interface in zoom and if you turn this on it will cause the zoom recordings to take up a lot
03:4803:48 - more dis space so for my purposes if I have a video that’s maybe 30 40 minutes long that will mean it’s kind of
03:5403:54 - hundreds of megabytes as opposed to tens um so definitely increases file size of the rec recordings but the videos are
04:0104:01 - presumably less compressed and will play nice with follow on video processing steps so I happen to do a lot of my
04:0804:08 - video processing with a library called ffmpeg but this would probably hold true for people who use I don’t know things
04:1504:15 - like Adobe Premiere Pro or D ventry resolve or any of the other uh what I would call more like normal uh uh video
04:2404:24 - editors um presumably those will you would want to do this if you edit your videos that way as well um so I think
04:3304:33 - it’s mostly to get kind of a standard frame rate a reduce compression you can do all of that in post rather than
04:3904:39 - having Zoom introduce any artifacts so uh this turned out to be exactly what I wanted and so uh my experience has borne
04:4704:47 - out that this setting actually does prevent you know The Unwanted behavior that I was getting before and so in the
04:5404:54 - test that I’ve run since I turned this on I haven’t actually had issues with my command line tool for removing silence
05:0005:00 - in the videos uh even though before it was kind of going crazy with the videos that I was getting out of Zoom like it
05:0505:05 - wasn’t working properly or the timing was all off or it was dropping sections and things like that U well with this
05:1205:12 - setting turned on and and the bigger files getting safe to dis with presumably less compression and whatever
05:1905:19 - whatever else they were doing to them that made them not play nice well with this setting turned on I actually
05:2405:24 - haven’t had any more problems uh integrating this with my automation on the command line since I’m using a
05:3105:31 - specific command line tool to do this now this is good news since it means that we won’t have to switch away from
05:3705:37 - zoom in our group recordings it turns out that the other options I was considering so things like Microsoft
05:4405:44 - teams Google meet things like that well these actually at present don’t allow for full screen screen recording since
05:5305:53 - uh they inject like a people list or participants list into the frame and you can’t get rid of it like you you can’t
05:5905:59 - turn it off in settings you can’t close out of it and so it just reduces uh what shows up in the recordings because if
06:0606:06 - you have a normal 16 by9 aspect ratio screen well now you have the sidebar in your screen recording and you can’t get
06:1306:13 - rid of it and so I find that very undesirable I don’t quite understand why these other uh platforms wouldn’t let
06:1906:19 - you record just your screen in the recording which is kind of irritating um but that is something very much in favor
06:2606:26 - of using Zoom for these group meetings because recording wise it leads to much better recording quality because you can
06:3306:33 - have it only focus on the content when you share your whole screen so for that reason we’d want to keep zoom and that’s
06:3906:39 - why I’m I’m thankful I managed to figure out how to make the zoom recordings work all right with the rest of my automated
06:4506:45 - video processing workflow uh because this way uh we get to keep the benefits of Zoom like this uh while also me being
06:5506:55 - able to uh automate all of the processing that I was previously having to do manually so at any rate um the
07:0207:02 - upshot of all this is that after determining that uh we can in fact keep using Zoom I wired up the silence
07:0807:08 - removal step in this command line tool that I’ve been working on to help me process the videos and so now all of
07:1507:15 - these or sorry rather this uh silence removal script will get automatically run rather than me having to manually
07:2207:22 - process each segment one by one like I was having to do before and so that should save me a lot of time on every
07:2807:28 - new video now I just run something and it does it for all the videos rather than me having to do things more
07:3407:34 - manually like I was doing previously so next we’re going to talk about the automated topic transition
07:4407:44 - scaffolding this was kind of the second of the big goals that I had uh coming out of the last progress update and so
07:5207:52 - alongside wiring up the silence removal I also set up the part of my automated video processing workflow that AO
07:5907:59 - automatically generates topic transition segments so that’s those uh 3 second black slides that show up in between the
08:0708:07 - video segments uh to kind of transition from one topic to the other you know like show the title of what’s now being
08:1408:14 - talked about and give it a bit of an audio Gap as well uh so that there’s kind of like a smooth transition rather
08:2008:20 - than it being like really abrupt from one thing to the others so here’s kind of how I’m doing this at a high level
08:2608:26 - this is a little technical but I mean it doesn’t get too down into the weeds um so from the markdown file which is what
08:3308:33 - I write my content in um I automatically parse out the headers that show up in the markdown file into an Excel
08:4008:40 - spreadsheet which I talked about in the previous Ministry progress update video kind of explained how that works in the
08:4608:46 - recordings uh process that I’m using and then I add any other miscellaneous headers that I need to like the intro
08:5208:52 - and outline one or the summary and outro one uh for just like General kind of administrative things and then using
09:0009:00 - that list in this Excel file I automatically build an HTML slido presentation uh uses this this package
09:0809:08 - here called remark um this is a uh basically a JavaScript utility that takes markdown formatted markdown and
09:1609:16 - turns it into HTML slido presentations and so I use that Library here to make slides out of all of the headers um that
09:2709:27 - will be video internal so what that means is that the very first section won’t have a transition slide because
09:3309:33 - I’m not going to start the video with a transition slide right but all the other ones there’ll be this nice transition
09:3909:39 - from section to section so once I have that slideshow presentation I turn it into a series of 1920 by 1080 so that’s
09:4809:48 - 1080p PNG images using a npm package called dect tape so this was a bit interesting for me to set up because
09:5609:56 - this is actually a JavaScript package um and my uh processing application is written in Python so I uh you know I am
10:0510:05 - calling this via you know a shell execute command in Python like I’m not directly importing it into python or
10:1110:11 - whatever because you can’t mix languages like that but it works fine you know I just have to figure out how to set it up
10:1510:15 - in npm and and get an npm environment set up so that I can use this as well uh when I’m running things on my computer U
10:2310:23 - but this basically takes the slideshow presentation and then turns it into a series of images
10:3010:30 - and then I turn those images into video files using FFM Peg that’s the video processing tool I’ve mentioned uh
10:3810:38 - several times before and so I uh need to ensure that the video files are 1920 x 1080 you know that’s the dimensions of
10:4610:46 - the images and so that’s the dimensions of the video as well and it also needs to be 25 frames per second those are
10:5310:53 - really important variables for lining up video files when you’re trying to concatenate them all together you have
10:5910:59 - to match resolution and frame rate so I am creating these myself and so I’m matching resolution and frame rate and
11:0611:06 - I’m turning them into 3 second long video files and so these are the ones that will go in between all of the
11:1111:11 - content sections and then the last thing you have to do is you have to add an audio track to these um because when
11:1811:18 - you’re using ffmpeg to stick all the videos together um there’s this thing called the concat filter here and when
11:2611:26 - you use this it looks something like this which probably won’t mean very much to a lot of people um but when you are
11:3211:32 - using the filter complex you have to align the video and audio streams and basically it only works if you have an
11:4011:40 - audio stream and so for these topic transitions there is no audio you know like it’s it’s I’m not saying anything
11:4611:46 - it’s going to be silent transitions but I have to add an audio track so that when I’m sticking everything together
11:5111:51 - that command will work properly and so basically uh to summarize all that really briefly so we get the headers and
11:5911:59 - then we build slides with the headers and then we turn the slides into images and then we turn the images into video
12:0412:04 - files and after all of that um that is the automatic creation of the topic transition segments and so those are the
12:1212:12 - things that go in between the recording segments and doing all of this in an automatic fashion means that I don’t
12:1912:19 - have to do anything manually so before how I was doing this was I was manually recording each of these segments and
12:2412:24 - then I was using a script to uh standardize them all to the same 3 second length but I was still having to
12:2912:29 - manually record each one and now I’m building these dynamically with the software tool and so I don’t have to do
12:3712:37 - anything I just run something and all of this happens behind the scenes automatically all right so next going to
12:4612:46 - be talking about automated transcripts so this actually wasn’t one of the things on my list I can’t recall if I
12:5312:53 - ever mentioned it in the last progress update but um this is another thing that was a little bit closer to being
13:0013:00 - completely automated so I was using regular expressions and a rather sophisticated spreadsheet a Google sheet
13:0713:07 - that used some regular expression extraction and stuff uh to do uh kind of what I am now doing in Python code uh
13:1513:15 - which is grabbing the transcript lines and um basically uh turning them into a specific format of short code so that I
13:2413:24 - can use a page internal YouTube links on my website I I think you can see the last progress update where I go over how
13:3213:32 - I implemented that in JavaScript um well I basically decided that I can do that in code and that saves me the hassle of
13:4213:42 - having to uh process stuff manually with regular expressions and then paste it into a spreadsheet so I was already
13:4813:48 - using um some tools to reduce most of the manual labor here um but now uh I just run the command just like the other
13:5713:57 - things and it automatically pulls down the transcript off of a YouTube video and sticks it all together in the format
14:0514:05 - that I need and puts it in the markdown content file so that when I have my web pages people can expand the transcript
14:1214:12 - section and search through it if they want to jump to specific Parts in the video so the library that I’m using here
14:1814:18 - it’s called YouTube transcript API this one is fortunately actually python rather than the other one which was
14:2414:24 - actually an npm package which was JavaScript um well because this is python I can it natively um now it is
14:3114:31 - relying on B it’s not tring about the right way to say this like a shadow API um it’s an undocumented API this is how
14:3914:39 - YouTube works like you hit this endpoint you get these things but it isn’t like a a stable API so there is the possibility
14:4814:48 - in the future that this project might no longer work the same way you know might have a breaking change if YouTube ever
14:5414:54 - just decides to update the thing that this is depending on in the background which so far hasn’t seemed to have
15:0115:01 - happened and you know this has been around for I don’t know a while now um so I’m optimistic that this will
15:0715:07 - continue to work but uh just kind of random gwiz information but Point here is that now I also have transcripts
15:1415:14 - completely automated so once I upload a video to YouTube YouTube will automatically generate a transcript I’m
15:2115:21 - assuming it uses some form of voice recognition uh AI to you know figure out what the words are that you’re saying
15:2815:28 - you know has been a thing for a while although I’m sure the the AI is getting better and things like that um so it
15:3415:34 - automatically generates transcripts uh to match the audio of the video file and so it is time like it is uh time matched
15:4315:43 - and then I can download that information and combine it um using this library and then stick it in my own webpage file so
15:5115:51 - that people can search the transcripts on my site and so the change here is that previously I was doing this sort of
15:5615:56 - like half automated and now it is completely automated just like the other things all right so at this point you
16:0616:06 - might be asking well what’s the big deal what is the upshot of all of this and so people who have been longtime followers
16:1316:13 - of this ministry probably know that I have been talking about kind of getting into content production mode for literal
16:2016:20 - years now you know years plural um I have been saying you know I will start making more content more consistently
16:2716:27 - once I get all of this process stuff done and so that was all this video processing things uh that I’ve been
16:3416:34 - talking about here and honestly talking about for a couple years it’s finally all coming together and so that has sort
16:4116:41 - of always been in the way from me giving myself permission to kind of shift the focus completely onto content because I
16:4816:48 - didn’t want to you know start making a lot of content only to have to go back later and and um you know dig myself out
16:5516:55 - of a hole of applying these these changes to it uh so that if I wanted to have time stamps or topic transitions
17:0317:03 - between things i’ have to go re-edit stuff in the future I wanted to have everything in place first before I go
17:1017:10 - crazy on the content side so that I don’t have any processing that I am kind of causing for myself in the future um
17:1717:17 - so regardless of the nitty-gritty details here the point of everything I’ve just gone through is that there
17:2317:23 - actually isn’t any more process of holding me back on the video content anymore the three things that I just
17:2817:28 - went over automating the silence removal part of it and automating the generation of
17:3317:33 - topic transition video segments and automating the transcript those were the last three things that I really have I
17:4017:40 - mean there are certainly some other refinements I could make so for example right now I’m still uploading the video
17:4617:46 - file to YouTube by hand I mean that takes all of a minute you know like I go to YouTube I upload the video I copy
17:5117:51 - paste the title and the description uh because I actually do automatically generate the description um as part of
17:5817:58 - my library so I just copy paste that from a text file and I mean it takes me only a minute or so but hypothetically
18:0318:03 - you could automate that too um but basically I’ve gotten rid of everything in the process that is actually timec
18:1018:10 - consuming in any way and so because of that um I think I am now in a position where I can finally throw that full
18:1718:17 - content production switch to on now that’s speaking from the video perspective if I were only doing videos
18:2318:23 - given what I’ve just finished I’d be done but that kind of brings us to the third goal uh that we were discussing
18:3118:31 - from the end of last video you know what were my short-term goals well the third one was getting the podcast launched and
18:3818:38 - so now that I’m mostly done with the video automation side of things I kind of need to work on the podcast
18:4418:44 - automation side of things and so that uh you know doing much of the similar you know adding a timestamp metadata to the
18:5118:51 - MP3 files that I’ll upload to the podcast hosting platforms things like that I me you know and just hosting and
18:5718:57 - syndicating the podcast more generally um but fortunately I think all of this should be a lot easier than the video
19:0419:04 - side of things because most of the processing steps ought to be pretty Apples to Apples here so if I’ve already
19:0919:09 - done it for the videos doing it again for the audio should be pretty easy um I mean there are some differences I’m sure
19:1619:16 - but um hosting this is really the thing that I have to figure out and making sure that um the timestamps and the
19:2119:21 - descriptions for each podcast episode play nice with all of the uh podcast websites like apple podcast and Spotify
19:2819:28 - but um I still need to figure this out and again that’s because if I put this off
19:3419:34 - and I just make the video content for everything well moving forward in the future I’m going to have this huge
19:3919:39 - backlog of stuff I’d have to do and I want to kind of have everything in place so that before I start making lots and
19:4419:44 - lots of content every new piece of content I do I will do all of this at the time that I make it and that way I
19:5119:51 - won’t ever get behind um so I’m really hoping that I can knock this piece out rapidly so that we can get out of this
19:5819:58 - limbo phase that we’ve been in for a while now um I am taking time off work for a couple weeks in the near future
20:0420:04 - early July here um to visit family and historically for me my quote unquote vacation time when I’m taking time off
20:1220:12 - from my day job has actually always been when I’ve gotten the most software development work done in any given year
20:1720:17 - so I have you know several projects up on GitHub maybe four or five and I mean you could check this well maybe people
20:2420:24 - wouldn’t know I know when I look at the dates that most of the times when I have a lot of commits you know if i’ if I’ve
20:3020:30 - done a lot of work on something it always lines up when when I was on vacation basically so I mean so much for
20:3720:37 - rest and relaxation on vacation right but because I will have free time when I’m not focused on my day job um I am
20:4520:45 - optimistic that I’ll be able to knock a lot of this out in the near future here so I mean I’m hoping if I buckle down
20:5120:51 - for maybe a week or something um I can get everything automated on the podcast side and figure out how
20:5920:59 - to basically do that last step in the process so that after uploading the video file I can upload the audio file
21:0521:05 - to the podcast host and everything else should just work and if I can get to that point then I’ll have to work
21:1221:12 - through the backlog of the Bible basics content um that we have already recorded so there’s 20 some odd videos there most
21:2021:20 - of those are up on YouTube and the ones that aren’t um we’ll get to in a second here um like next section in this update
21:2821:28 - um but but basically once I do that I will really be able to consistently move forward for each additional new piece of
21:3521:35 - content I make and that kind of brings us to the last main part of this update which is
21:4321:43 - talking about the content that I will be consistently posting in the next little bit here so uh that podcast automation
21:5121:51 - aside that we just talked about since I’ve now finished the video side of things I’m basically good to go for
21:5621:56 - getting through the backlog of recorded video content that we have and getting everything posted to YouTube and the
22:0322:03 - website so that would include things like the subject index links and tagging the the headers with passages and things
22:0922:09 - like that um I’ve made a pretty good dent on this already um so uh I will be releasing a video a day here or
22:1722:17 - thereabouts for the next couple weeks at least so I probably already got about 10 that are ready to go I’m just going to
22:2222:22 - space them out day by day and then I have maybe another 10 that I need to actually go finish um but now that I’ve
22:2822:28 - finished the automation or you know for example automatically generating the transition slides and pulling everything
22:3422:34 - together then I can actually go finish this backlog of videos because I’d sort of been holding off on finishing
22:4122:41 - everything until I had the automation so that I wouldn’t waste time um kind of from a Time efficiency perspective and
22:4722:47 - so most of this content is from the group study that we had on Bible basics part one theology um there’s also a
22:5422:54 - separate four-part video playlist in there uh from when a friend ask me some questions and uh this particular
23:0123:01 - playlist will fall into the content type on the site corresponding to questions and answers reader correspondence so um
23:0823:08 - it ended up being kind of long actually you know a couple of those videos in that series are more than an hour um and
23:1323:13 - I think it’s it’s interesting stuff it’s about resuscitation and the afterlife and sleep as a euphemism for death and
23:1923:19 - things like that um but that content will also go up here in the next couple weeks as I post through the backlog of
23:2623:26 - content that I already have recorded and so this is something that I’ve been promising for a while as well I said
23:3123:31 - once I get all that Automation in place I’ll go ahead and post the backlog of stuff well thus far I haven’t posted the
23:3823:38 - backlog of stuff because I haven’t finished the automation but now that I’ve done that um I really will get
23:4223:42 - through this and so it’ll probably be at least 15 videos over the next couple weeks here um all going up in the Bible
23:4923:49 - basic series and this other playlist I mentioned and that will get us to the point where I’m basically caught up from
23:5423:54 - a video perspective all the stuff that we’ve recorded will now be live on the website on YouTube and then once I
24:0024:00 - finish the podcast stuff we will truly be caught up and then I’ll be ready to get going on all the new
24:0824:08 - stuff and so that kind of brings us to this last section that I always do on these Ministry updates and that’s kind
24:1424:14 - of what am I planning on doing in the future here and short term I’m really going to try to put my head down and get
24:2024:20 - the podcast launched because I want to finish that before I tackle anything else and like I said I’m hopeful that
24:2724:27 - that will happen when I’m on vacation over the next several weeks but if not I I do still want this to be my top
24:3424:34 - priority because once I do this uh we can launch uh the new group study from the zoom Bible study for readers of IUS
24:4324:43 - if I launch inperson stuff at my house I’ll be able to keep up uh with each new week and in-person studies like I’ll
24:4924:49 - really be able to throw that content production switch like we’ve been talking about um and this is kind of the
24:5424:54 - last gatekeeper of that so this is really what I’m going to be focusing on finishing in the next little bit
25:0225:02 - here and so that’s it for this one um I know not so much on the content side here although like I mentioned I will be
25:1025:10 - consistently posting the backlog of stuff that we’ve already done and uh for a lot of people maybe this wasn’t that
25:1625:16 - interesting but for the long-term long uh sustainability of the content of this ministry I think a lot of the things
25:2425:24 - that kind of got done here will really help make this week by week much more sustainable um so getting that automated
25:3225:32 - silence removal the transition segment scaffolding the transcripts completely automated uh getting the podcast set up
25:3925:39 - as that last thing I have to do once I get all of this in place then every new week I will be able to create two or
25:4625:46 - three new videos uh two or three new podcast episodes you know which is the same as the videos but just the audio
25:5325:53 - I’m like I’ll be able to do that and keep up a pace of maybe three or four videos a week which would be complet
25:5825:58 - compl impossible if I had to manually do anything with those videos I you know I do work a full-time job and I’m sure I’m
26:0426:04 - only going to get busier as I have other obligations so to make that possible I really have to automate a lot of things
26:1026:10 - and so getting all of this in place is that’s what I’m shooting for here um is getting that two or three or four videos
26:1726:17 - a week um getting everything I need set up so that I can do that sustainably in the Long Haul and so I’m excited because
26:2526:25 - we’re close and this is uh you know it’s very much ress I’ve been working on this stuff for quite a long time now and
26:3126:31 - having it all come together is really encouraging so stay tuned um I hope in the next update here I will I will be
26:3826:38 - bearing news that I got the podcast set up and then you’ll hear about my next plans after that but that will hopefully
26:4526:45 - be uh what gets covered in the next one