Creating, Writing, and Reading a file in production environment

I have ruby on rails app that also uses python to provide functions to one of its features which is pose estimation. What it does is it compares the dance moves of one user, to the other dance moves of another user in a form of video. The thing is, the video is being passed by my rails app, to python script using sys.argv. After passing the video to the python script, it will do its thing with pose estimation, and the structure on how it provide the result of the dance move comparison to my rails app, is by creating a temporary file where the result data can be written. Then going back to my rails app, it will now read the content of the temporary file, then present it’s value to the user with the web app interface.

With that, I’m having an issue because in the runtime logs, it says that the file does not exist, which I assume that it’s either creating a temporary file in the deployed environment isn’t possible, or I’m just passing the incorrect directory to where the temporary file could possibly exist upon creation. Either ways, I’ll be providing the configuration of my rails app, and my python script which allows it to work in local environment but not in deployed environment(which is the the problem of mine I’m trying to raise here). I’ve been working on this project for quite a while now, and this is the last part that I need to finish, so hopefully someone can help me.

For the Configuration
This is the method in my rails controller that triggers the python script and pass the necessary video files for it to compare.

  def run_script
    @class_recital = ClassRecital.find(params[:id])
  
    # Creating temp file for instructor video
    temp_instructor_video = Tempfile.new(["instructor_video", ".mp4"])
    File.open(temp_instructor_video.path, 'wb') do |f|
      f.write(DanceClass.find(@class_recital.class_id).choreography.download)
    end
    instructor_video_path = temp_instructor_video.path.gsub('\\', '/')
  
    # Creating temp file for student video
    temp_student_video = Tempfile.new(["student_video", ".mp4"])
    File.open(temp_student_video.path, 'wb') do |f|
      f.write(@class_recital.recital.download)
    end
    student_video_path = temp_student_video.path.gsub('\\', '/')
  
    python_path = "python3"
    script_path = Rails.root.join("t1.py").to_s
  
    # Execute the Python script, passing in the paths
    result = `#{python_path} #{script_path} "#{instructor_video_path}" "#{student_video_path}"`

    # After running the Python script, read the result from temp_result.txt
    begin
      temp_result_file = "/data/temp_result.txt"
      performance_score = File.read("temp_result.txt").to_f.round(2)
      @class_recital.update(performance_score: performance_score)
    rescue => e
      Rails.logger.error "Error reading temp result file: #{e.message}"
    ensure
      # Clean up the temp_result.txt file
      File.delete("temp_result.txt") if File.exist?("temp_result.txt")
    end
  
    # Close and unlink the temp files
    temp_instructor_video.close
    temp_instructor_video.unlink
    temp_student_video.close
    temp_student_video.unlink
  
    # Respond to the client
    respond_to do |format|
      format.js
      format.html { redirect_to @class_recital, notice: "Analysis Completed. Check the server log for results." }
    end
  end

Lastly, this is the section of my python script where it receives the video files from the rails app, trigger the comparison function, then create a file to store the results so that the rails app can read it.

#.......Beginning of the code........

# Initialize video capture for the two videos to be compared
cap_instructor = cv2.VideoCapture(r"{}".format(sys.argv[1]))
cap_student = cv2.VideoCapture(r"{}".format(sys.argv[2]))

#........Rest of the code......

#lastly, file creation and writing process
    result_file_path = "/data/temp_result.txt"
    with open("temp_result.txt", "w") as f:
        f.write(str(avg_percentage))

Having the exact error message would help, but based on the excerpts you have provided my first guess would be that the problem isn’t that the file doesn’t exist, but that the /data directory doesn’t exist. That’s not a directory that normally would exist unless you explicitly took an action to create it.

I see, for the actual error message, here it is

2023-12-11T21:48:51.746 app[3d8d907c374389] sin [info] 2023-12-11 21:48:51.746058: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.

2023-12-11T21:48:51.746 app[3d8d907c374389] sin [info] To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

2023-12-11T21:49:10.601 app[3d8d907c374389] sin [info] Traceback (most recent call last):

2023-12-11T21:49:10.601 app[3d8d907c374389] sin [info] File "/rails/t1.py", line 140, in <module>

2023-12-11T21:49:10.601 app[3d8d907c374389] sin [info] with open("temp_result.txt", "w") as f:

2023-12-11T21:49:10.601 app[3d8d907c374389] sin [info] PermissionError: [Errno 13] Permission denied: 'temp_result.txt'

2023-12-11T21:49:12.324 app[3d8d907c374389] sin [info] [76013070-f5c0-4887-b9ed-918d78a85d45] Error reading temp result file: No such file or directory @ rb_sysopen - temp_result.txt

2023-12-11T21:49:12.325 app[3d8d907c374389] sin [info] [76013070-f5c0-4887-b9ed-918d78a85d45] Redirected to https://danceclass.fly.dev/class_recitals/1

2023-12-11T21:49:12.325 app[3d8d907c374389] sin [info] [76013070-f5c0-4887-b9ed-918d78a85d45] Completed 302 Found in 21436ms (ActiveRecord: 7.1ms | Allocations: 2242)

Look in your Dockerfile for lines that look like the following:

Make sure that /data is on the line that starts with chown.

That 's presuming that the python script is being run out of the /data directory, which is not clear from your example.

I’m sorry if the info I provided is not clear. Regarding that, my app does not run from the /data directory. Upon troubleshooting, I just assumed that upon creating a new temporary file, is that it might be stored in the destination as assigned in the mount configuration when creating and attaching a volume to an app. Originally, I was using Rails.root.join(‘temp_file.txt’).to_s where I assumed that it might be stored in the root directory of my app upon creation, just like how it works in the local environment. With that, it’s the first instance that I encountered the error, right before I started trying out other stuff to solve it.

And regarding the example you provided that I should look for in my docker file, I think that does not exist in my docker file. The docker file I’m using is the default docker generated upon using fly launch. It’s only modified to take care of the python environment for my python script and its dependencies.

If you need more information about my docker configuration, here’s what my docker file looks like:

# syntax = docker/dockerfile:1

ARG RUBY_VERSION=3.1.2
FROM ruby:$RUBY_VERSION-slim as base

LABEL fly_launch_runtime="rails"

WORKDIR /rails

ENV RAILS_ENV="production" \
    BUNDLE_WITHOUT="development:test" \
    BUNDLE_DEPLOYMENT="1"

RUN gem update --system --no-document && \
    gem install -N bundler

# Install Python 3.10 and pip
RUN apt-get update -qq && \
    apt-get install -y --no-install-recommends python3.10 python3-pip && \
    rm -rf /var/lib/apt/lists/*

# Install Python dependencies
COPY requirements.txt ./
RUN pip3 install --no-cache-dir -r requirements.txt

FROM base as build

RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential libpq-dev libvips pkg-config

COPY --link Gemfile Gemfile.lock ./
RUN bundle install && \
    bundle exec bootsnap precompile --gemfile && \
    rm -rf ~/.bundle/ $BUNDLE_PATH/ruby/*/cache $BUNDLE_PATH/ruby/*/bundler/gems/*/.git

COPY --link . .

RUN bundle exec bootsnap precompile app/ lib/ && \
    chmod +x bin/* && \
    sed -i "s/\r$//g" bin/* && \
    sed -i 's/ruby\.exe$/ruby/' bin/* && \
    SECRET_KEY_BASE=DUMMY ./bin/rails assets:precompile

FROM base

RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl imagemagick libsqlite3-0 libvips postgresql-client && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

COPY --from=build /usr/local/bundle /usr/local/bundle
COPY --from=build /rails /rails

RUN useradd rails --create-home --shell /bin/bash && \
    chown -R rails:rails db log storage tmp
USER rails:rails

ENV RAILS_LOG_TO_STDOUT="1" \
    RAILS_SERVE_STATIC_FILES="true"

ENTRYPOINT ["/rails/bin/docker-entrypoint"]
EXPOSE 3000
CMD ["./bin/rails", "server"]

Also, as a person who’s self-studying and is still foreign to these concerns and concepts, your response is greatly appreciated. Thank you.

This sets the current working directory. Absent any other instructions, that’s where files will be put. That directory is owned by root.

This creates a new user, changes the ownership of 4 specific subdirectories to be owned by that user, and then changes to become that user.

Circling back to your original Python code:

If you were to change that code so that the file was placed into one of the four named subdirectories, the write would succeed. For example:

    with open("tmp/temp_result.txt", "w") as f:
        f.write(str(avg_percentage))

Upon trying out these changes, I still encountered the same error.

In the python script:

    with open("tmp/temp_result.txt", "w") as f:
        f.write(str(avg_percentage))

In the run_script method responsible for triggering the python script:

# After running the Python script, read the result from temp_result.txt
    begin
      temp_result_file = "tmp/temp_result.txt"
      performance_score = File.read("temp_result.txt").to_f.round(2)
      @class_recital.update(performance_score: performance_score)

Something must be changing the current working directory. No problem, we can be more explicit:

    with open("/rails/tmp/temp_result.txt", "w") as f:
        f.write(str(avg_percentage))

That should likely be:

temp_result_file = "/rails/tmp/temp_result.txt"
performance_score = File.read(temp_result_file).to_f.round(2)

Note the last line.

With that configuration, it is working now. Again, I’m really thankful for the help.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.